AutoGen with Local LLMs: Setup with Ollama and vLLM

Introduction to AutoGen with Local LLMs

AutoGen is an open-source framework by Microsoft for building multi-agent conversational AI applications. It enables LLM-powered agents to communicate, use tools, execute code, and collaborate on complex tasks. Typically, AutoGen relies on cloud-based APIs like OpenAI or Azure. However, running local large language models (LLMs) via Ollama or vLLM gives you full privacy, eliminates per-token costs, removes rate limits, and allows offline operation. This tutorial covers the complete setup for using AutoGen with local models, from installation to building a functional multi-agent system.

What is AutoGen and Why Local LLMs?

AutoGen provides high-level abstractions for agents (AssistantAgent, UserProxyAgent, GroupChatManager) that can converse, plan, and execute code. Local LLMs bring several advantages:

Data privacy: All conversations and code stay on your machine.
Cost control: No API bills — you only pay for electricity and hardware.
No rate limits: Unlimited requests for development and testing.
Offline capability: Works without internet access after model download.
Full model control: Choose any open model, fine-tune, or quantize as needed.

Ollama is a lightweight, easy-to-use local model runner that supports many open models out of the box. vLLM is a high-performance inference server designed for production workloads, offering continuous batching, PagedAttention, and an OpenAI-compatible API. Both integrate seamlessly with AutoGen via its flexible LLM configuration.

Prerequisites and Installation

We'll set up AutoGen, Ollama (for quick local inference), and optionally vLLM (for production-grade serving). Ensure you have Python 3.10+ and pip installed.

Installing Ollama

Install Ollama on Linux/macOS with a single command:

curl -fsSL https://ollama.ai/install.sh | sh

Alternatively, download the installer from the official website for your platform. After installation, pull a model like Llama 3:

ollama pull llama3

Ollama serves the model on http://localhost:11434 by default, exposing an OpenAI-compatible endpoint at /v1.

Installing vLLM (Optional)

vLLM provides a dedicated OpenAI-compatible API server. Install it via pip:

pip install vllm

To serve a model (e.g., Llama 3 8B), run:

python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3-8B-Instruct \
    --host 0.0.0.0 --port 8000

This makes the model available at http://localhost:8000/v1. vLLM requires a compatible GPU; for CPU-only setups, stick with Ollama.

Installing AutoGen

pip install pyautogen

AutoGen relies on the openai Python client to communicate with local endpoints:

pip install openai

Configuring AutoGen for Local LLMs

AutoGen uses a config_list dictionary to specify model endpoints. You can point it to any OpenAI-compatible server, including Ollama and vLLM. The key parameters are model, base_url, and api_key. For local models, cost tracking should be disabled by setting "price": [0, 0].

Using Ollama with AutoGen

Create a configuration for Ollama. The api_key can be any non-empty string (it is not validated by Ollama).

import autogen

# Ollama config – model name as pulled (e.g., "llama3")
config_list_ollama = [
    {
        "model": "llama3",
        "base_url": "http://localhost:11434/v1",
        "api_key": "ollama",
        "price": [0, 0]   # disable cost tracking
    }
]

# Create an assistant agent
assistant_ollama = autogen.AssistantAgent(
    name="assistant",
    llm_config={
        "config_list": config_list_ollama,
        "temperature": 0.7,
        "max_tokens": 1024,
    }
)

The base_url points to Ollama's /v1 endpoint. This configuration works for any model you've pulled with Ollama — just change the "model" value accordingly.

Using vLLM with AutoGen

vLLM's API is fully OpenAI-compatible. Use the exact model identifier you passed to vLLM.

config_list_vllm = [
    {
        "model": "meta-llama/Llama-3-8B-Instruct",
        "base_url": "http://localhost:8000/v1",
        "api_key": "EMPTY",   # vLLM ignores the key
        "price": [0, 0]
    }
]

assistant_vllm = autogen.AssistantAgent(
    name="assistant_vllm",
    llm_config={
        "config_list": config_list_vllm,
        "temperature": 0.2,
        "max_tokens": 2048,
    }
)

Building a Simple Multi-Agent Conversation

Let's build a classic AutoGen setup: an AssistantAgent that generates responses using a local model, and a UserProxyAgent that simulates a user, can execute code, and provides feedback.

from autogen import AssistantAgent, UserProxyAgent

# Assistant using Ollama config from previous step
assistant = AssistantAgent(
    name="assistant",
    llm_config={
        "config_list": config_list_ollama,
        "temperature": 0.2,
    },
    system_message="You are a helpful AI assistant. Reply succinctly and provide correct Python code."
)

# User proxy with code execution enabled
user_proxy = UserProxyAgent(
    name="user",
    human_input_mode="NEVER",        # fully automated
    max_consecutive_auto_reply=3,    # limit back-and-forth
    code_execution_config={
        "work_dir": "coding",        # directory for code files
        "use_docker": False          # run locally, not in container
    }
)

# Start the conversation
user_proxy.initiate_chat(
    assistant,
    message="Write a Python function to compute the factorial of a number and test it with 5."
)

The assistant generates a function, the user proxy executes it, and they discuss the output. All inference happens locally through Ollama.

Using GroupChat with Local Models

For more complex scenarios, AutoGen supports group chats where a manager agent (powered by an LLM) decides who speaks next. You can use a local model as the group chat manager.

from autogen import GroupChat, GroupChatManager

# Create a group chat with the previously defined agents
groupchat = GroupChat(
    agents=[user_proxy, assistant],
    messages=[],
    max_round=5
)

# Manager uses the same Ollama config
manager = GroupChatManager(
    groupchat=groupchat,
    llm_config={"config_list": config_list_ollama}
)

# Start a task that requires reasoning and code
user_proxy.initiate_chat(
    manager,
    message="Solve the quadratic equation x^2 - 4x + 4 = 0 using Python, and explain the roots."
)

The manager orchestrates the conversation, deciding when the assistant should speak or when the user proxy should run code.

Handling Tool Use and Function Calling

Local models vary in their support for native function calling. With AutoGen, you have several strategies:

Code-based tools: The assistant generates Python code; the user proxy executes it as a tool.
Native function calling: If the model supports it (e.g., Mistral 7B, Llama 3.1, or fine-tuned variants), you can register functions directly.

Below is an example of registering a custom function with Ollama using a model that supports tool calls.

# Define a tool function
def get_weather(city: str) -> str:
    # In a real scenario, call a weather API
    return f"The weather in {city} is sunny, 72°F."

# Assistant configuration with function_map
assistant_with_tools = autogen.AssistantAgent(
    name="tool_agent",
    system_message="You must use the provided 'get_weather' function when asked about weather.",
    llm_config={
        "config_list": [
            {
                "model": "mistral",          # or "llama3.1" etc.
                "base_url": "http://localhost:11434/v1",
                "api_key": "ollama",
                "price": [0, 0]
            }
        ],
        "temperature": 0.0,
    },
    function_map={
        "get_weather": get_weather
    }
)

# User proxy that forwards function calls
user_proxy_tool = autogen.UserProxyAgent(
    name="tool_executor",
    human_input_mode="NEVER",
    code_execution_config=False
)

user_proxy_tool.initiate_chat(
    assistant_with_tools,
    message="What is the weather in Paris?"
)

Note: Native function calling requires the model to output properly formatted JSON tool calls. If your local model does not reliably do this, fall back to code execution: let the assistant write a Python script that calls the function, and let the user proxy run it.

Best Practices and Tips

Choose the right model: For coding tasks, use CodeLlama, DeepSeek-Coder, or Llama 3. For general conversation, Mistral or Llama 3 work well. Verify the model's context length and instruction-following quality.
Disable cost tracking: Always set "price": [0, 0] in your config_list entries. This prevents AutoGen from trying to estimate token costs, which would fail for local endpoints.
Set max_consecutive_auto_reply: Limit the number of automatic replies to avoid infinite loops. A value of 3–5 is usually safe.
Monitor resource usage: Running LLMs locally can be GPU/CPU intensive. Use quantized models (e.g., Ollama's GGUF variants) to reduce memory footprint. For vLLM, adjust --max-model-len and --gpu-memory-utilization.
Use vLLM for production, Ollama for development: vLLM delivers high throughput and continuous batching, while Ollama is simpler to set up and debug. Start with Ollama, then switch to vLLM when you need performance.
Provide a fallback to cloud APIs: Include an OpenAI or Azure entry in your config_list as a backup. AutoGen will try local models first and fall back to cloud if the local endpoint fails.
Test connectivity first: Before running complex agents, send a minimal request to your local endpoint using curl or the OpenAI Python client to ensure it's responding correctly.

Advanced: Integrating with External Tools and LangChain

AutoGen agents can seamlessly use tools from LangChain, custom Python functions, or APIs. With local models, the most reliable pattern is to let the assistant plan and generate code, while the user proxy executes it. This works independently of the model's native function-calling ability.

# Example: download a file and count lines
assistant_planner = autogen.AssistantAgent(
    name="planner",
    llm_config={"config_list": config_list_ollama},
    system_message="Plan tasks and write Python code. The executor will run the code and return results."
)

executor = autogen.UserProxyAgent(
    name="executor",
    human_input_mode="NEVER",
    code_execution_config={"work_dir": "workspace"}
)

executor.initiate_chat(
    assistant_planner,
    message="Download the README from https://raw.githubusercontent.com/microsoft/autogen/main/README.md and count the number of lines."
)

The assistant will produce a script using requests (or urllib), the executor runs it, and they iterate until the task is complete.

Conclusion

Combining AutoGen with local LLMs via Ollama and vLLM unlocks a private, cost-effective, and highly controllable environment for building multi-agent AI systems. You can prototype entirely offline, avoid API rate limits, and keep sensitive data on-premises. The setup is straightforward thanks to OpenAI-compatible APIs: just point AutoGen's config to http://localhost:11434/v1 (Ollama) or http://localhost:8000/v1 (vLLM). By following the patterns in this tutorial — from basic conversations to group chats and tool integration — you can create powerful agent workflows that run entirely on your own hardware. Experiment with different open models, tune your prompts, and enjoy the freedom of local inference.