← Back to DevBytes
Developer Tutorial July 19, 2026 15 min read

AutoGen with Local LLMs: Setup with Ollama and vLLM

Introduction to AutoGen with Local LLMs

AutoGen is Microsoft's open-source framework for building conversational multi-agent systems powered by large language models. It enables developers to create complex agent workflows where multiple AI agents collaborate, debate, and execute tasks autonomously. While AutoGen works seamlessly with cloud-based APIs like OpenAI and Azure OpenAI, running it with local LLMs opens up a new world of possibilities — complete data privacy, zero token costs, offline capability, and full control over model selection and configuration.

This tutorial walks you through setting up AutoGen with two leading local LLM serving solutions: Ollama (optimized for ease of use and desktop-class inference) and vLLM (designed for high-throughput production serving). By the end, you'll have a fully functional multi-agent system running entirely on your own hardware.

Why Run AutoGen with Local LLMs?

There are several compelling reasons to pair AutoGen with locally hosted models:

Prerequisites

Before diving in, ensure you have the following:

Create a dedicated project directory and virtual environment to keep dependencies isolated:

mkdir autogen-local-llms
cd autogen-local-llms
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Setting Up AutoGen with Ollama

Ollama provides a frictionless way to download, manage, and serve LLMs locally with an OpenAI-compatible API. It's ideal for development workstations and single-user scenarios.

Step 1: Install and Launch Ollama

Install Ollama using the official script or package manager for your OS. On Linux/macOS:

curl -fsSL https://ollama.com/install.sh | sh

On Windows, download the installer from ollama.com. Once installed, start the Ollama server (it typically runs as a background service, but you can also launch it manually):

ollama serve

The server listens on http://localhost:11434 by default. Verify it's running by visiting http://localhost:11434 in your browser — you should see "Ollama is running".

Step 2: Pull a Model

Download a model that suits your hardware and task requirements. For a good balance of capability and resource usage, Meta's Llama 3.1 8B or Mistral 7B are excellent choices:

ollama pull llama3.1:8b
# Or for a smaller footprint:
ollama pull mistral:7b

Verify the model is available:

ollama list

You can test the model directly from the command line:

ollama run llama3.1:8b "Explain the concept of multi-agent AI systems in one paragraph."

Step 3: Install AutoGen with Ollama Support

Install the core AutoGen library along with the Ollama-specific extensions:

pip install pyautogen
pip install litellm  # Provides Ollama integration layer

Note that litellm acts as a translation layer, mapping various LLM providers (including Ollama's API) into a unified format that AutoGen understands.

Step 4: Configure AutoGen to Use Ollama

AutoGen uses a config_list dictionary to specify which LLM endpoints to use. For Ollama, you configure it as an OpenAI-compatible endpoint pointing to the local server. Here's a complete working example of a two-agent conversation:

import autogen

# Configure the LLM to point to your local Ollama server
config_list = [
    {
        "model": "llama3.1:8b",
        "api_key": "ollama",  # Ollama doesn't require a real key, but a placeholder is needed
        "base_url": "http://localhost:11434/v1",
        "api_type": "openai",  # Ollama's API is OpenAI-compatible
    }
]

# Create the assistant agent (the LLM-powered agent)
assistant = autogen.AssistantAgent(
    name="assistant",
    llm_config={
        "config_list": config_list,
        "temperature": 0.7,
        "max_tokens": 1024,
        "timeout": 120,
    },
    system_message="You are a helpful AI assistant. Provide clear, concise responses.",
)

# Create the user proxy agent (acts on behalf of the human user)
user_proxy = autogen.UserProxyAgent(
    name="user_proxy",
    human_input_mode="NEVER",  # Fully automated mode
    max_consecutive_auto_reply=5,
    code_execution_config={
        "work_dir": "coding_workdir",
        "use_docker": False,  # Set to True if you have Docker installed
    },
)

# Initiate the conversation
user_proxy.initiate_chat(
    assistant,
    message="Write a Python function that calculates the Fibonacci sequence up to n terms using recursion, and include a test case for n=10.",
)

print("\nConversation complete! Check 'coding_workdir' for generated files.")

This script creates two agents: an AssistantAgent backed by your local Llama 3.1 model via Ollama, and a UserProxyAgent that can execute code. The conversation runs fully automated — the assistant writes code, and the user proxy can execute it and report results back.

Step 5: Advanced Ollama Configuration

For more granular control, you can specify additional parameters that Ollama supports. Here's an enhanced configuration example:

# Enhanced configuration with Ollama-specific options
config_list = [
    {
        "model": "llama3.1:8b",
        "api_key": "ollama",
        "base_url": "http://localhost:11434/v1",
        "api_type": "openai",
        # Additional parameters passed through litellm
        "api_version": None,  # Not used by Ollama but required by litellm schema
        "num_retries": 3,  # Auto-retry on failures
        "request_timeout": 300,  # Longer timeout for complex generations
    }
]

# You can also specify model-specific generation parameters
llm_config = {
    "config_list": config_list,
    "temperature": 0.3,  # Lower temperature for more deterministic coding tasks
    "max_tokens": 2048,
    "top_p": 0.9,
    "frequency_penalty": 0.1,
    "presence_penalty": 0.1,
    "timeout": 180,
    "seed": 42,  # For reproducible outputs
}

assistant = autogen.AssistantAgent(
    name="precise_assistant",
    llm_config=llm_config,
    system_message="You are an expert Python developer. Write clean, well-documented code with proper error handling.",
)

Setting Up AutoGen with vLLM

vLLM is a high-performance LLM serving engine designed for production workloads. It features PagedAttention for efficient memory management, continuous batching, and supports tensor parallelism across multiple GPUs. Use vLLM when you need maximum throughput, want to serve multiple concurrent users, or are deploying in a server environment.

Step 1: Install vLLM

vLLM requires a GPU with CUDA support. Install it via pip:

pip install vllm

For systems with multiple GPUs or specific CUDA versions, consult the vLLM documentation. You can verify the installation:

python -c "import vllm; print(vllm.__version__)"

Step 2: Download a Model for vLLM

vLLM works with models from Hugging Face. Popular choices include:

For models requiring authentication, log in to Hugging Face CLI:

pip install huggingface_hub
huggingface-cli login
# Paste your Hugging Face token when prompted

Step 3: Launch the vLLM Server

Start vLLM with an OpenAI-compatible API endpoint. Here's the command to serve Mistral 7B on a single GPU:

python -m vllm.entrypoints.openai.api_server \
    --model mistralai/Mistral-7B-Instruct-v0.3 \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 4096 \
    --gpu-memory-utilization 0.9 \
    --dtype auto

Key parameters explained:

For multi-GPU setups, add tensor parallelism:

# Serve on 2 GPUs with tensor parallelism
python -m vllm.entrypoints.openai.api_server \
    --model mistralai/Mistral-7B-Instruct-v0.3 \
    --tensor-parallel-size 2 \
    --host 0.0.0.0 \
    --port 8000

The server will take 30-60 seconds to load the model (depending on size and disk speed). Once ready, you'll see output indicating the API is available at http://localhost:8000.

Step 4: Verify the vLLM Server

Test the endpoint with a simple curl request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-7B-Instruct-v0.3",
    "messages": [{"role": "user", "content": "Hello! What is 2+2?"}],
    "max_tokens": 50
  }'

You should receive a JSON response with the model's answer.

Step 5: Configure AutoGen for vLLM

The configuration is nearly identical to Ollama since both expose OpenAI-compatible APIs. Here's a complete multi-agent setup using vLLM:

import autogen

# vLLM server configuration
config_list = [
    {
        "model": "mistralai/Mistral-7B-Instruct-v0.3",
        "api_key": "EMPTY",  # vLLM doesn't require authentication by default
        "base_url": "http://localhost:8000/v1",
        "api_type": "openai",
    }
]

# Create a specialized coding assistant
coding_assistant = autogen.AssistantAgent(
    name="coder",
    llm_config={
        "config_list": config_list,
        "temperature": 0.2,
        "max_tokens": 2048,
        "timeout": 300,
        "seed": 123,
    },
    system_message="""You are an expert software engineer.
    When asked to write code:
    1. First explain your approach briefly
    2. Write clean, well-commented code
    3. Include error handling
    4. Add a usage example or test case""",
)

# Create a reviewer agent that critiques the code
reviewer = autogen.AssistantAgent(
    name="reviewer",
    llm_config={
        "config_list": config_list,
        "temperature": 0.5,
        "max_tokens": 1024,
        "timeout": 300,
    },
    system_message="""You are a code reviewer. After the coder produces code:
    1. Check for correctness and edge cases
    2. Suggest improvements for readability and performance
    3. Verify the code follows best practices
    Be constructive and specific in your feedback.""",
)

# User proxy to manage the workflow
user_proxy = autogen.UserProxyAgent(
    name="user_proxy",
    human_input_mode="NEVER",
    max_consecutive_auto_reply=10,
    code_execution_config={
        "work_dir": "vllm_workspace",
        "use_docker": False,
    },
)

# Create a group chat with both agents
groupchat = autogen.GroupChat(
    agents=[user_proxy, coding_assistant, reviewer],
    messages=[],
    max_round=8,
    speaker_selection_method="auto",
)

manager = autogen.GroupChatManager(
    name="manager",
    groupchat=groupchat,
    llm_config={
        "config_list": config_list,
        "temperature": 0.1,
        "max_tokens": 512,
        "timeout": 120,
    },
)

# Launch the collaborative session
user_proxy.initiate_chat(
    manager,
    message="""I need a Python implementation of a task scheduler that:
    1. Accepts tasks with priorities and deadlines
    2. Uses a priority queue
    3. Supports adding, removing, and listing tasks
    4. Includes thread-safe operations
    Please have the coder implement it and the reviewer provide feedback.""",
)

print("\nCollaborative session complete!")

This example demonstrates a group chat pattern where a coding agent and a reviewer agent collaborate under the management of a group chat coordinator. The manager automatically selects which agent should speak next based on conversation context, creating a natural collaborative workflow entirely powered by your local vLLM instance.

Handling Common Issues

Ollama Troubleshooting

vLLM Troubleshooting

Best Practices for Local LLM + AutoGen Deployments

1. Match Model Capability to Agent Role

Not all agents need the most powerful model. Consider using different models for different agents in the same workflow:

# Use a larger model for complex reasoning tasks
planner_config = {
    "config_list": [{
        "model": "llama3.1:70b",  # Powerful model for planning
        "base_url": "http://localhost:11434/v1",
        "api_key": "ollama",
        "api_type": "openai",
    }],
    "temperature": 0.7,
}

# Use a smaller, faster model for straightforward execution
executor_config = {
    "config_list": [{
        "model": "llama3.1:8b",  # Smaller model for code execution
        "base_url": "http://localhost:11434/v1",
        "api_key": "ollama",
        "api_type": "openai",
    }],
    "temperature": 0.1,
}

planner = autogen.AssistantAgent(name="planner", llm_config=planner_config)
executor = autogen.AssistantAgent(name="executor", llm_config=executor_config)

2. Implement Robust Error Handling

Local LLMs can occasionally produce malformed outputs or timeout. Wrap your AutoGen runs in try-except blocks:

import time
import sys

def run_agent_workflow_with_retry(max_retries=3):
    for attempt in range(max_retries):
        try:
            user_proxy.initiate_chat(
                assistant,
                message="Your task description here",
            )
            return True
        except Exception as e:
            print(f"Attempt {attempt + 1} failed: {e}")
            if attempt < max_retries - 1:
                wait_time = 2 ** attempt * 10  # Exponential backoff
                print(f"Retrying in {wait_time} seconds...")
                time.sleep(wait_time)
            else:
                print("All retry attempts exhausted.")
                raise
    return False

run_agent_workflow_with_retry()

3. Optimize for Your Hardware

Configure generation parameters based on your available resources:

4. Leverage Streaming for Better UX

AutoGen supports streaming responses. Enable it to see agent outputs in real-time rather than waiting for the full response:

llm_config = {
    "config_list": config_list,
    "temperature": 0.7,
    "max_tokens": 2048,
    "stream": True,  # Enable streaming
    "timeout": 300,
}

assistant = autogen.AssistantAgent(
    name="streaming_assistant",
    llm_config=llm_config,
)

5. Cache Model Responses During Development

When iterating on agent logic, cache LLM responses to avoid redundant inference calls:

# Enable caching in the config
llm_config = {
    "config_list": config_list,
    "temperature": 0,
    "max_tokens": 1024,
    "use_cache": True,  # Cache identical requests
}

# You can also specify a custom cache seed for reproducibility
assistant = autogen.AssistantAgent(
    name="cached_assistant",
    llm_config=llm_config,
)

# Clear cache between major changes
assistant.client.cache = None  # Reset the cache

6. Monitor Resource Usage

Keep an eye on GPU memory and CPU utilization when running extended agent sessions. For vLLM, use the built-in metrics endpoint:

# Enable metrics in vLLM
python -m vllm.entrypoints.openai.api_server \
    --model mistralai/Mistral-7B-Instruct-v0.3 \
    --host 0.0.0.0 \
    --port 8000 \
    --enable-metrics

# Query metrics (in a separate terminal)
curl http://localhost:8000/metrics

For Ollama, monitor with system tools like nvidia-smi or htop.

7. Use Structured Outputs When Possible

Guide local models to produce well-formatted responses by crafting detailed system messages:

system_message = """You are a data extraction agent.
Always respond in valid JSON format with the following structure:
{
    "summary": "Brief summary of findings",
    "details": ["key point 1", "key point 2"],
    "confidence": 0.0 to 1.0
}
Do not include any text outside the JSON block.
Wrap the JSON in json code fences."""

extractor = autogen.AssistantAgent(
    name="extractor",
    llm_config=llm_config,
    system_message=system_message,
)

Performance Comparison: Ollama vs vLLM

Here's a practical comparison to help you choose the right backend:

For most AutoGen development workflows, start with Ollama for rapid iteration, then move to vLLM if you need production-grade performance or are running multiple concurrent agent sessions.

Complete End-to-End Example: Code Review Pipeline

Let's tie everything together with a complete, production-ready example that works with either Ollama or vLLM. This pipeline implements an automated code generation, review, and improvement cycle:

import autogen
import os
import json
from datetime import datetime

# ============================================
# CONFIGURATION - Switch between Ollama and vLLM
# ============================================

USE_OLLAMA = True  # Set to False for vLLM

if USE_OLLAMA:
    # Ollama configuration
    BASE_URL = "http://localhost:11434/v1"
    CODING_MODEL = "llama3.1:8b"
    REVIEW_MODEL = "llama3.1:8b"
    API_KEY = "ollama"
else:
    # vLLM configuration
    BASE_URL = "http://localhost:8000/v1"
    CODING_MODEL = "mistralai/Mistral-7B-Instruct-v0.3"
    REVIEW_MODEL = "mistralai/Mistral-7B-Instruct-v0.3"
    API_KEY = "EMPTY"

# Shared configuration template
def create_agent_config(model, temperature=0.3, max_tokens=2048):
    return {
        "config_list": [{
            "model": model,
            "api_key": API_KEY,
            "base_url": BASE_URL,
            "api_type": "openai",
        }],
        "temperature": temperature,
        "max_tokens": max_tokens,
        "timeout": 300,
        "stream": True,
    }

# ============================================
# AGENT DEFINITIONS
# ============================================

# Coder agent - generates initial implementations
coder = autogen.AssistantAgent(
    name="SeniorDeveloper",
    llm_config=create_agent_config(CODING_MODEL, temperature=0.2, max_tokens=2048),
    system_message="""You are a senior software engineer with 15 years of experience.
    Your task is to write production-quality Python code based on requirements.
    
    Guidelines:
    - Write complete, working implementations (not stubs)
    - Include comprehensive docstrings
    - Add type hints for all function signatures
    - Handle edge cases and errors gracefully
    - Include unit tests as part of your response
    - Output code in python code blocks""",
)

# Reviewer agent - provides constructive feedback
reviewer = autogen.AssistantAgent(
    name="CodeReviewer",
    llm_config=create_agent_config(REVIEW_MODEL, temperature=0.4, max_tokens=1500),
    system_message="""You are a meticulous code reviewer. After reviewing code, provide:
    
    1. Overall assessment (score 1-10)
    2. Specific issues found (if any)
    3. Suggested improvements with code examples
    4. Security concerns (if applicable)
    5. Performance considerations
    
    Be thorough but constructive. If the code is excellent, say so clearly.
    Format your review in structured markdown.""",
)

# Refiner agent - improves code based on feedback
refiner = autogen.AssistantAgent(
    name="CodeRefiner",
    llm_config=create_agent_config(CODING_MODEL, temperature=0.1, max_tokens=2048),
    system_message="""You are a code refinement specialist.
    Given original code and review feedback, produce an improved version that:
    
    - Addresses ALL issues raised in the review
    - Maintains backward compatibility
    - Adds any missing error handling
    - Improves performance where suggested
    - Output the complete refined code in python blocks""",
)

# User proxy agent
user_proxy = autogen.UserProxyAgent(
    name="ProjectManager",
    human_input_mode="NEVER",
    max_consecutive_auto_reply=15,
    code_execution_config={
        "work_dir": "code_review_workspace",
        "use_docker": False,
    },
    system_message="Project manager overseeing the code generation and review process.",
)

# ============================================
# ORCHESTRATION LOGIC
# ============================================

def run_code_review_pipeline(task_description):
    """
    Orchestrates a complete code generation → review → refinement cycle.
    """
    
    # Phase 1: Generate initial code
    print("\n" + "="*60)
    print("PHASE 1: Code Generation")
    print("="*60)
    
    generation_result = user_proxy.initiate_chat(
        coder,
        message=f"""Please implement the following requirement:
        
        {task_description}
        
        Provide the complete implementation with tests.""",
        clear_history=True,
    )
    
    # Extract the generated code (last message from coder)
    coder_response = user_proxy.last_message()
    
    # Phase 2: Review the code
    print("\n" + "="*60)
    print("PHASE 2: Code Review")
    print("="*60)
    
    review_prompt = f"""Please review the following code implementation:

{coder_response.get('content', str(coder_response))}

Provide a thorough code review following your review guidelines."""
    
    review_result = user_proxy.initiate_chat(
        reviewer,
        message=review_prompt,
        clear_history=True,
    )
    
    reviewer_response = user_proxy.last_message()
    
    # Phase 3: Refine based on feedback
    print("\n" + "="*60)
    print("PHASE 3: Code Refinement")
    print("="*60)
    
    refinement_prompt = f"""Original code:
{coder_response.get('content', str(coder_response))}

Review feedback:
{reviewer_response.get('content', str(reviewer_response))}

Please produce a refined version of the code addressing all feedback."""
    
    refinement_result = user_proxy.initiate_chat(
        refiner,
        message=refinement_prompt,
        clear_history=True,
    )
    
    refiner_response = user_proxy.last_message()
    
    # Save results
    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    output_dir = f"pipeline_output_{timestamp}"
    os.makedirs(output_dir, exist_ok=True)
    
    with open(f"{output_dir}/original_code.md", "w") as f:
        f.write(str(coder_response.get('content', '')))
    
    with open(f"{output_dir}/review.md", "w") as f:
        f.write(str(reviewer_response.get('content', '')))
    
    with open(f"{output_dir}/refined_code.md", "w") as f:
        f.write(str(refiner_response.get('content', '')))
    
    print(f"\nPipeline complete! Output saved to {output_dir}/")
    
    return {
        "original": coder_response,
        "review": reviewer_response,
        "refined": refiner_response,
        "output_dir": output_dir,
    }

# ============================================
# RUN THE PIPELINE
# ============================================

if __name__ == "__main__":
    task = """
    Create a Python class called 'RateLimiter' that implements a sliding window
    rate limiting algorithm. Requirements:
    
    1. Constructor accepts: max_requests (int), window_seconds (float)
    2. Method 'is_allowed(client_id: str) -> bool' that returns True if the
       client hasn't exceeded the limit in the current window
    3. Thread-safe implementation using appropriate locking
    4. Efficient cleanup of expired entries
    5. Support for multiple concurrent clients
    
    Example usage should be included in the tests.
    """
    
    results = run_code_review_pipeline(task)
    
    print("\nFinal refined code:")
    print(results["refined"].get("content

— Ad —

Google AdSense will appear here after approval

← Back to all articles