Structured Output Validation with AutoGen: Complete Guide

Introduction to Structured Output Validation with AutoGen

AutoGen, Microsoft's multi-agent conversation framework, has become a popular choice for building LLM-powered applications. One of the most powerful features it offers is structured output validation — the ability to constrain and verify that model responses conform to a predefined schema. This tutorial walks you through everything you need to know to use structured output validation effectively in your AutoGen projects.

What Is Structured Output Validation?

Structured output validation is the process of forcing a language model to produce responses that match a specific data structure, such as a JSON object with defined fields, types, and constraints. Instead of parsing free-form text and hoping the model cooperates, you declare a schema upfront and the framework enforces it.

In AutoGen, this is typically achieved through integration with Pydantic models or JSON schemas. The framework validates the model's output against your schema and can automatically retry or provide feedback when validation fails.

Why It Matters

• Reliability: Eliminates fragile regex parsing and ad-hoc string manipulation.
• Type Safety: Catches type mismatches and missing fields before they propagate through your application.
• Composability: Structured outputs can be directly fed into downstream functions, databases, or APIs.
• Reduced Latency: Fewer retries and less post-processing means faster end-to-end execution.
• Better Testing: Schemas make it trivial to write assertions and integration tests.

Setting Up Your Environment

Before diving into code, install AutoGen and its dependencies. The examples in this tutorial use AutoGen 0.4+ (the AG2 / autogen-agentchat package) along with Pydantic v2.

pip install "autogen-agentchat[openai]" pydantic

Set your OpenAI API key as an environment variable:

export OPENAI_API_KEY="your-api-key-here"

For local development, you can also use a .env file with python-dotenv:

from dotenv import load_dotenv
load_dotenv()

Defining Your Schema with Pydantic

The foundation of structured output validation is a well-defined schema. Pydantic models are the idiomatic way to declare these in Python. Let's create a schema for extracting structured information from a product review.

from pydantic import BaseModel, Field
from typing import Literal, Optional
from datetime import date


class ProductReview(BaseModel):
    product_name: str = Field(..., description="The name of the product being reviewed")
    rating: int = Field(..., ge=1, le=5, description="Rating from 1 to 5")
    sentiment: Literal["positive", "neutral", "negative"] = Field(
        ..., description="Overall sentiment of the review"
    )
    summary: str = Field(..., max_length=200, description="A concise summary of the review")
    recommended: bool = Field(..., description="Whether the reviewer recommends the product")
    tags: list[str] = Field(default_factory=list, description="Key topics mentioned in the review")
    purchase_date: Optional[date] = Field(None, description="Date of purchase if mentioned")

Notice how each field uses Field to provide descriptions and constraints. These descriptions are passed to the model as part of the prompt, which dramatically improves compliance. Constraints like ge=1, le=5 and max_length=200 are enforced by Pydantic during validation.

Basic Usage with AutoGen Agents

AutoGen provides a clean API for requesting structured output from an assistant agent. The create method on AssistantAgent accepts a response_format parameter where you pass your Pydantic model.

import asyncio
from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient


async def main():
    model_client = OpenAIChatCompletionClient(
        model="gpt-4o-2024-08-06",
        temperature=0.3,
    )

    agent = AssistantAgent(
        name="review_analyzer",
        model_client=model_client,
        system_message=(
            "You are a product review analyst. Extract structured information "
            "from user-provided reviews. Be precise and only include information "
            "explicitly stated in the review."
        ),
    )

    review_text = """
    I bought the Sony WH-1000XM5 headphones on March 15, 2024 and I'm blown away.
    The noise cancellation is incredible and the battery lasts forever. I'd give
    it 5 stars and absolutely recommend it to anyone in the market for premium
    headphones. Worth every penny.
    """

    result = await agent.run(
        task=f"Analyze this review:\n\n{review_text}",
        response_format=ProductReview,
    )

    # The structured output is available on the last message
    print(result.messages[-1].content)


asyncio.run(main())

When you pass response_format, AutoGen instructs the underlying model to produce JSON conforming to your schema. The response is then parsed and validated automatically. If the model returns invalid JSON or a value that violates your constraints, AutoGen will raise a validation error.

Handling Validation Failures with Retries

In production, models occasionally produce outputs that fail validation. A robust application should handle these cases gracefully. AutoGen allows you to wrap your agent calls in retry logic and feed validation errors back to the model so it can self-correct.

import asyncio
from pydantic import ValidationError
from autogen_agentchat.agents import AssistantAgent
from autogen_ext.models.openai import OpenAIChatCompletionClient


async def extract_with_retry(agent, task, schema, max_retries=3):
    messages = [task]
    
    for attempt in range(max_retries):
        result = await agent.run(task="\n\n".join(messages))

        raw_output = result.messages[-1].content

        try:
            parsed = schema.model_validate_json(raw_output)
            return parsed
        except ValidationError as e:
            error_feedback = (
                f"Your previous output failed validation with these errors:\n"
                f"{e}\n\n"
                f"Please regenerate the output as valid JSON matching the schema. "
                f"Return ONLY the JSON, no additional text."
            )
            messages.append(raw_output)
            messages.append(error_feedback)
            print(f"Attempt {attempt + 1} failed, retrying...")

    raise RuntimeError(f"Failed to produce valid output after {max_retries} attempts")


async def main():
    model_client = OpenAIChatCompletionClient(model="gpt-4o-2024-08-06")
    agent = AssistantAgent(
        name="extractor",
        model_client=model_client,
        system_message="You extract structured data. Always respond with valid JSON only.",
    )

    task = "Extract a ProductReview from: 'The widget is okay, 3 stars, not sure if I'd recommend it.'"
    
    review = await extract_with_retry(agent, task, ProductReview)
    print(review.model_dump_json(indent=2))


asyncio.run(main())

This pattern is powerful because the model sees exactly what went wrong and can adjust. In practice, most failures are resolved on the second attempt.

Using JSON Schema Directly

If you prefer not to use Pydantic, or if your schema is generated dynamically, you can pass a raw JSON schema dictionary. This is useful when schemas are loaded from configuration files or external services.

review_schema = {
    "type": "object",
    "properties": {
        "product_name": {"type": "string"},
        "rating": {"type": "integer", "minimum": 1, "maximum": 5},
        "sentiment": {
            "type": "string",
            "enum": ["positive", "neutral", "negative"]
        },
        "summary": {"type": "string", "maxLength": 200},
        "recommended": {"type": "boolean"},
        "tags": {
            "type": "array",
            "items": {"type": "string"}
        }
    },
    "required": ["product_name", "rating", "sentiment", "summary", "recommended"],
    "additionalProperties": False
}

result = await agent.run(
    task="Analyze this review...",
    response_format=review_schema,
)

The additionalProperties: False setting is important — it prevents the model from inventing extra fields that could cause issues downstream.

Structured Output in Multi-Agent Workflows

One of AutoGen's strengths is multi-agent orchestration. Structured outputs become especially valuable when agents need to hand data to each other. Let's build a pipeline where one agent extracts a review, another fact-checks it, and a third generates a response.

import asyncio
from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_agentchat.conditions import TextMentionTermination
from autogen_ext.models.openai import OpenAIChatCompletionClient


class FactCheckResult(BaseModel):
    claims_verified: int
    claims_unverified: int
    notes: str
    is_accurate: bool


class CustomerResponse(BaseModel):
    greeting: str
    acknowledgment: str
    action_taken: str
    closing: str


async def main():
    model_client = OpenAIChatCompletionClient(model="gpt-4o-2024-08-06")

    extractor = AssistantAgent(
        name="extractor",
        model_client=model_client,
        system_message="Extract a ProductReview from the customer's message.",
    )

    fact_checker = AssistantAgent(
        name="fact_checker",
        model_client=model_client,
        system_message=(
            "You receive a product review. Verify the claims are reasonable. "
            "Respond with a FactCheckResult JSON object."
        ),
    )

    responder = AssistantAgent(
        name="responder",
        model_client=model_client,
        system_message=(
            "Draft a professional customer service response based on the review "
            "and fact check. Respond with a CustomerResponse JSON object. "
            "End your message with TERMINATE when done."
        ),
    )

    team = RoundRobinGroupChat(
        participants=[extractor, fact_checker, responder],
        termination_condition=TextMentionTermination("TERMINATE"),
        max_turns=6,
    )

    result = await team.run(
        task="Customer review: 'My Acme Blender broke after 2 weeks. 1 star. Not recommended.'",
    )

    for msg in result.messages:
        print(f"[{msg.source}]: {msg.content}\n")


asyncio.run(main())

In this workflow, each agent produces structured output that the next agent can parse and reason about. This eliminates ambiguity in inter-agent communication and makes the entire pipeline more debuggable.

Advanced Validation Techniques

Custom Validators

Pydantic supports custom validators for business logic that goes beyond simple type checking. Use @field_validator and @model_validator decorators to enforce complex rules.

from pydantic import BaseModel, Field, field_validator, model_validator


class Order(BaseModel):
    order_id: str
    items: list[str]
    quantities: list[int]
    total_price: float
    currency: str = "USD"

    @field_validator("order_id")
    @classmethod
    def validate_order_id(cls, v):
        if not v.startswith("ORD-"):
            raise ValueError("order_id must start with 'ORD-'")
        return v

    @field_validator("quantities")
    @classmethod
    def no_negative_quantities(cls, v):
        if any(q <= 0 for q in v):
            raise ValueError("All quantities must be positive")
        return v

    @model_validator(mode="after")
    def check_items_quantities_length(self):
        if len(self.items) != len(self.quantities):
            raise ValueError("items and quantities must have the same length")
        return self

Nested Models

Real-world schemas often involve nested structures. Pydantic handles these naturally, and AutoGen passes the full nested schema to the model.

class Address(BaseModel):
    street: str
    city: str
    state: str
    zip_code: str


class Customer(BaseModel):
    name: str
    email: str
    shipping_address: Address
    billing_address: Optional[Address] = None
    phone: Optional[str] = Field(None, pattern=r"^\+?[\d\s\-\(\)]+$")

Discriminated Unions

When a response can be one of several types, use discriminated unions. This is common in routing or classification tasks.

from typing import Annotated, Literal, Union


class BugReport(BaseModel):
    type: Literal["bug"] = "bug"
    severity: Literal["low", "medium", "high", "critical"]
    steps_to_reproduce: list[str]


class FeatureRequest(BaseModel):
    type: Literal["feature"] = "feature"
    priority: Literal["nice_to_have", "important", "critical"]
    use_case: str


class Question(BaseModel):
    type: Literal["question"] = "question"
    topic: str


SupportTicket = Annotated[
    Union[BugReport, FeatureRequest, Question],
    Field(discriminator="type")
]

When you pass SupportTicket as your response_format, the model first determines the type field and then produces the appropriate structure.

Best Practices

Write Descriptive Field Descriptions

The model uses field descriptions to understand what to produce. Vague descriptions lead to poor results. Compare these two definitions:

# Poor
class Bad(BaseModel):
    name: str
    value: float

# Good
class Good(BaseModel):
    name: str = Field(..., description="Full legal name of the customer, including middle name if applicable")
    value: float = Field(..., description="Total order value in USD, including tax and shipping, rounded to 2 decimal places")

Use Enums and Literals for Constrained Values

Whenever a field can only take a fixed set of values, use Literal or enum.Enum. This prevents the model from inventing values and makes validation strict.

Keep Schemas Focused

Avoid creating massive schemas with dozens of fields. Large schemas confuse models and increase the chance of validation failures. If you need complex output, break it into multiple agent calls, each producing a smaller, focused structure.

Set Temperature Low

Structured output extraction is a deterministic task. Use a low temperature (0.0–0.3) to reduce randomness and improve consistency.

model_client = OpenAIChatCompletionClient(
    model="gpt-4o-2024-08-06",
    temperature=0.1,
)

Always Handle the Failure Case

Even with the best schemas, validation will occasionally fail in production. Implement retry logic, log failures for analysis, and consider a fallback strategy such as defaulting to a safe value or escalating to a human.

Test Your Schemas Independently

Write unit tests that validate sample outputs against your Pydantic models. This catches schema issues early, before they interact with the model.

import pytest
from pydantic import ValidationError


def test_valid_review():
    data = {
        "product_name": "Test Product",
        "rating": 4,
        "sentiment": "positive",
        "summary": "Great product overall.",
        "recommended": True,
        "tags": ["quality", "value"],
    }
    review = ProductReview.model_validate(data)
    assert review.rating == 4


def test_invalid_rating_rejected():
    data = {
        "product_name": "Test Product",
        "rating": 10,
        "sentiment": "positive",
        "summary": "Great product.",
        "recommended": True,
    }
    with pytest.raises(ValidationError):
        ProductReview.model_validate(data)

Leverage Model-Specific Structured Output Modes

Modern models like GPT-4o support native structured output modes (JSON Schema / function calling). AutoGen leverages these under the hood when you pass response_format. Using a model that supports native structured outputs yields significantly better compliance than relying on prompt-based JSON instructions alone.

Common Pitfalls and How to Avoid Them

• Optional fields treated as required: If you use Optional[T] without a default, Pydantic still requires the key to be present (with value null). Use Optional[T] = None to make the key truly optional.
• Strict mode surprises: Pydantic v2 in strict mode rejects strings that look like numbers and vice versa. Be aware of this when the model produces loosely typed JSON.
• Forgetting additionalProperties: False: Without this, the model may add unexpected fields. In Pydantic, use model_config = ConfigDict(extra="forbid").
• Overly complex nested schemas: Deep nesting (4+ levels) degrades model performance. Flatten where possible.

from pydantic import BaseModel, ConfigDict


class StrictModel(BaseModel):
    model_config = ConfigDict(extra="forbid", strict=True)
    
    name: str
    count: int

Conclusion

Structured output validation is one of the most impactful patterns for building reliable LLM applications with AutoGen. By defining clear Pydantic schemas, leveraging AutoGen's response_format parameter, implementing retry logic for validation failures, and following best practices around schema design and model configuration, you can transform unpredictable free-text responses into dependable, typed data that integrates seamlessly with the rest of your system. Start simple with a focused schema, test it thoroughly, and iterate as your multi-agent workflows grow in complexity. The investment in structured outputs pays dividends in reliability, debuggability, and developer confidence from day one.