Building a Code Review Agent with Pydantic AI: Complete Guide

Building a Code Review Agent with Pydantic AI: Complete Guide

Code reviews are one of the most valuable yet time-consuming activities in modern software development. With the rise of structured LLM frameworks, we can now build agents that perform meaningful, consistent code analysis at scale. Pydantic AI, a framework that pairs Pydantic's type validation with LLM orchestration, is particularly well-suited for this task because it enforces structured outputs and makes agent behavior predictable.

In this tutorial, you'll build a fully functional Code Review Agent that accepts source code, analyzes it against configurable rules, and returns structured, actionable feedback. By the end, you'll understand how to design agents that produce reliable, typed outputs suitable for integration into CI/CD pipelines, IDE extensions, or developer tools.

What Is Pydantic AI?

Pydantic AI is a Python agent framework built by the team behind Pydantic. It lets you define agents that call LLMs and return validated, typed Python objects rather than free-form text. The framework supports multiple model providers (OpenAI, Anthropic, Gemini, Ollama, and others) and emphasizes type safety, dependency injection, and structured tool use.

Why It Matters for Code Review

• Predictable outputs: Reviews come back as typed objects, not unstructured prose you have to parse.
• Validation by default: Pydantic ensures the model's response matches your schema, catching hallucinated fields.
• Provider-agnostic: Swap models without rewriting agent logic.
• Dependency injection: Pass repositories, linters, or configuration cleanly into tools.
• Testability: Mock models and dependencies for deterministic unit tests.

Project Setup

Create a new project directory and install the required dependencies. We'll use OpenAI as the default provider, but the code is structured so you can switch easily.

mkdir code-review-agent
cd code-review-agent
python -m venv .venv
source .venv/bin/activate

pip install pydantic-ai pydantic python-dotenv

Create a .env file with your API key:

OPENAI_API_KEY=sk-your-key-here

Create the project structure:

code-review-agent/
├── .env
├── main.py
├── agent.py
├── models.py
└── sample_code.py

Defining the Output Schema

The foundation of a reliable agent is a well-defined output model. Instead of letting the LLM ramble, we constrain it to return a structured review with severity levels, line references, and concrete suggestions.

Create models.py:

from enum import Enum
from pydantic import BaseModel, Field
from typing import Optional


class Severity(str, Enum):
    INFO = "info"
    WARNING = "warning"
    ERROR = "error"
    CRITICAL = "critical"


class Issue(BaseModel):
    """A single issue found during code review."""
    line_number: Optional[int] = Field(
        None,
        description="Approximate line number where the issue occurs, if applicable."
    )
    severity: Severity = Field(
        ...,
        description="How impactful the issue is."
    )
    category: str = Field(
        ...,
        description="Short category like 'security', 'performance', 'style', 'bug', 'maintainability'."
    )
    description: str = Field(
        ...,
        description="Clear explanation of the issue."
    )
    suggestion: str = Field(
        ...,
        description="Concrete fix or improvement the developer can apply."
    )


class CodeReview(BaseModel):
    """The complete structured result of a code review."""
    summary: str = Field(
        ...,
        description="A 1-3 sentence overall assessment of the code."
    )
    overall_score: int = Field(
        ...,
        ge=0,
        le=10,
        description="Quality score from 0 (terrible) to 10 (excellent)."
    )
    issues: list[Issue] = Field(
        default_factory=list,
        description="List of issues found, ordered by severity (critical first)."
    )
    positive_aspects: list[str] = Field(
        default_factory=list,
        description="Things the code does well worth highlighting."
    )
    approved: bool = Field(
        ...,
        description="Whether the code would pass a human review."
    )

Notice how every field has a description. Pydantic AI uses these descriptions as part of the prompt sent to the model, which dramatically improves output quality.

Building the Agent

Now create agent.py. This is where we define the agent, its system prompt, and any tools it can use.

from pydantic_ai import Agent, RunContext
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.openai import OpenAIProvider
from models import CodeReview

import os
from dotenv import load_dotenv

load_dotenv()

model = OpenAIModel(
    "gpt-4o",
    provider=OpenAIProvider(api_key=os.getenv("OPENAI_API_KEY")),
)

code_review_agent = Agent(
    model=model,
    output_type=CodeReview,
    deps_type=None,
    system_prompt=(
        "You are a senior software engineer performing a thorough code review. "
        "You analyze code for bugs, security vulnerabilities, performance issues, "
        "maintainability problems, and style violations. "
        "Be specific and reference line numbers when possible. "
        "Provide actionable suggestions, not vague advice. "
        "Only mark a review as approved if the code has no errors or critical issues. "
        "Always respond in the structured format defined by the output schema."
    ),
)


@code_review_agent.tool
async def get_language_guidelines(ctx: RunContext[None], language: str) -> str:
    """Return best-practice guidelines for a given programming language."""
    guidelines = {
        "python": (
            "Follow PEP 8. Prefer type hints. Avoid mutable default arguments. "
            "Use context managers for resources. Prefer pathlib over os.path."
        ),
        "javascript": (
            "Prefer const/let over var. Use strict equality (===). "
            "Handle promises with async/await. Avoid global state."
        ),
        "typescript": (
            "Enable strict mode. Avoid 'any'. Use interfaces for object shapes. "
            "Leverage discriminated unions for predictable control flow."
        ),
    }
    return guidelines.get(language.lower(), "Follow general clean code principles.")


async def review_code(code: str, language: str = "python") -> CodeReview:
    """Run a code review and return the structured result."""
    prompt = (
        f"Review the following {language} code. "
        f"Use the get_language_guidelines tool to fetch best practices first.\n\n"
        f"\n{code}\n"
    )
    result = await code_review_agent.run(prompt)
    return result.output

How This Works

The Agent constructor takes three key arguments here: the model to use, the output_type (our CodeReview schema), and a system_prompt that defines the agent's persona. When output_type is a Pydantic model, Pydantic AI automatically instructs the model to produce JSON conforming to that schema and validates the response.

The @code_review_agent.tool decorator registers a function the agent can call during its reasoning. Here, the agent can fetch language-specific guidelines before reviewing. Tools are how you extend an agent's capabilities beyond pure text generation.

Running the Agent

Let's create a sample file with intentionally flawed code to review. Create sample_code.py:

def process_users(users=[]):
    results = []
    for i in range(len(users)):
        user = users[i]
        if user['active'] == True:
            password = "admin123"
            results.append({
                'name': user['name'],
                'password': password
            })
    return results

def get_data(url):
    import urllib.request
    data = urllib.request.urlopen(url).read()
    return data.decode()

This snippet contains several real issues: a mutable default argument, a hardcoded password, a boolean comparison with ==, an unused loop index, and a network call without error handling or timeouts.

Now create main.py to run the review:

import asyncio
from agent import review_code
from sample_code import process_users, get_data  # noqa: F401
import inspect


async def main():
    # Read the source code of the functions we want to review
    source = inspect.getsource(process_users) + "\n\n" + inspect.getsource(get_data)

    print("Reviewing code...\n")
    review = await review_code(source, language="python")

    print(f"Summary: {review.summary}")
    print(f"Score: {review.overall_score}/10")
    print(f"Approved: {'Yes' if review.approved else 'No'}\n")

    print("Positive aspects:")
    for aspect in review.positive_aspects:
        print(f"  + {aspect}")

    print("\nIssues found:")
    for issue in review.issues:
        line = f" (line {issue.line_number})" if issue.line_number else ""
        print(f"  [{issue.severity.value.upper()}] {issue.category}{line}")
        print(f"    {issue.description}")
        print(f"    Suggestion: {issue.suggestion}\n")


if __name__ == "__main__":
    asyncio.run(main())

Run it:

python main.py

You should see output similar to:

Reviewing code...

Summary: The code has several significant issues including a security vulnerability and a common Python anti-pattern. It requires fixes before approval.
Score: 4/10
Approved: No

Positive aspects:
  + Functions are small and focused
  + Return values are consistent

Issues found:
  [CRITICAL] security (line 7)
    Hardcoded password "admin123" is a major security risk.
    Suggestion: Use environment variables or a secrets manager to load credentials.

  [ERROR] bug (line 1)
    Mutable default argument `users=[]` will persist across calls.
    Suggestion: Use `users=None` and initialize inside the function.

  [WARNING] style (line 4)
    Comparison with True using `==` is unnecessary.
    Suggestion: Use `if user['active']:` directly.

  [WARNING] performance (line 2)
    Using range(len(users)) and indexing is non-idiomatic.
    Suggestion: Iterate directly with `for user in users:`.

  [WARNING] bug (line 15)
    Network call has no timeout or error handling.
    Suggestion: Use requests with a timeout, or wrap urllib in try/except.

Adding Dependency Injection

Real-world agents often need access to external resources: a Git repository, a linter, or a configuration object. Pydantic AI supports this through typed dependencies. Let's extend the agent to accept a configuration that controls which categories of issues to focus on.

Update agent.py to include a dependencies type:

from dataclasses import dataclass


@dataclass
class ReviewConfig:
    focus_categories: list[str]  # e.g. ["security", "bug"]
    strict_mode: bool = False
    max_issues: int = 20


code_review_agent = Agent(
    model=model,
    output_type=CodeReview,
    deps_type=ReviewConfig,
    system_prompt=(
        "You are a senior software engineer performing a thorough code review. "
        "Analyze code for bugs, security vulnerabilities, performance issues, "
        "maintainability problems, and style violations. "
        "Be specific and reference line numbers when possible. "
        "Provide actionable suggestions. "
        "If a focus_categories list is provided, prioritize those categories. "
        "In strict mode, never approve code with any warning or higher issue."
    ),
)


@code_review_agent.tool
async def get_review_config(ctx: RunContext[ReviewConfig]) -> str:
    """Return the current review configuration."""
    cfg = ctx.deps
    return (
        f"Focus categories: {', '.join(cfg.focus_categories)}. "
        f"Strict mode: {cfg.strict_mode}. "
        f"Max issues to report: {cfg.max_issues}."
    )


async def review_code(
    code: str,
    language: str = "python",
    config: ReviewConfig = None,
) -> CodeReview:
    prompt = (
        f"Review the following {language} code. "
        f"Check the review configuration with get_review_config.\n\n"
        f"\n{code}\n"
    )
    result = await code_review_agent.run(prompt, deps=config)
    return result.output

Now you can run a security-focused review:

config = ReviewConfig(focus_categories=["security"], strict_mode=True)
review = await review_code(source, language="python", config=config)

Streaming and Token Usage

For long reviews, you may want to stream output or track token usage. Pydantic AI supports both:

async def review_code_streamed(code: str, language: str = "python"):
    prompt = f"Review the following {language} code:\n\n\n{code}\n"
    async with code_review_agent.run_stream(prompt) as result:
        # Stream partial outputs as they arrive
        async for partial in result.stream(output_type=CodeReview):
            print(f"Partial: {partial.summary} ({len(partial.issues)} issues so far)")
        final = await result.get_output()
        return final

To inspect token usage after a run:

result = await code_review_agent.run(prompt)
print(f"Input tokens: {result.usage().input_tokens}")
print(f"Output tokens: {result.usage().output_tokens}")

Testing the Agent

One of Pydantic AI's strengths is testability. You can register a TestModel that returns canned responses without calling a real LLM. Create test_agent.py:

import pytest
from pydantic_ai.models.test import TestModel
from agent import code_review_agent, review_code
from models import CodeReview, Severity, Issue


@pytest.mark.asyncio
async def test_review_returns_structured_output(monkeypatch):
    test_model = TestModel()
    test_model.set_output(
        CodeReview(
            summary="Test summary",
            overall_score=7,
            issues=[
                Issue(
                    line_number=1,
                    severity=Severity.WARNING,
                    category="style",
                    description="Test issue",
                    suggestion="Fix it",
                )
            ],
            positive_aspects=["Clean structure"],
            approved=True,
        )
    )
    code_review_agent.model = test_model

    review = await review_code("def foo(): pass", language="python")
    assert review.overall_score == 7
    assert len(review.issues) == 1
    assert review.issues[0].category == "style"
    assert review.approved is True

Install pytest and run the tests:

pip install pytest pytest-asyncio
pytest test_agent.py -v

Best Practices

Design Schemas Carefully

Your output schema is the contract between the LLM and your application. Keep fields focused, use enums for constrained values, and always include descriptions. Avoid optional fields unless truly necessary — required fields force the model to commit to an answer.

Use System Prompts to Set Boundaries

Be explicit about what the agent should and should not do. Specify the severity thresholds for approval, the expected tone, and how to handle ambiguity. Vague system prompts produce vague reviews.

Provide Tools for Grounding

Tools let the agent fetch real data instead of guessing. For a code review agent, useful tools include language guideline lookups, access to a project's linting configuration, or the ability to read related files from a repository.

Validate Before Acting

Because Pydantic AI validates outputs against your schema, you can trust the structure. But still validate the content in your application logic — for example, ensure line numbers fall within the actual code range before displaying them to users.

Handle Failures Gracefully

LLM calls can fail due to rate limits, malformed outputs, or network issues. Wrap agent calls in try/except blocks and consider retry logic with exponential backoff for production use.

from pydantic_ai.exceptions import ModelHTTPError

async def safe_review(code: str, language: str = "python", retries: int = 3):
    for attempt in range(retries):
        try:
            return await review_code(code, language)
        except ModelHTTPError as e:
            if attempt == retries - 1:
                raise
            await asyncio.sleep(2 ** attempt)

Log and Monitor

Track token usage, latency, and approval rates over time. This helps you identify when a model change degrades review quality or when certain code patterns consistently confuse the agent.

Keep Humans in the Loop

An AI code review agent is a first-pass filter, not a replacement for human review. Use it to catch obvious issues and surface them early, but always have a human make the final approval decision on significant changes.

Conclusion

Building a code review agent with Pydantic AI gives you a powerful, type-safe way to automate a high-value development activity. By defining a clear output schema, crafting a focused system prompt, and leveraging tools and dependency injection, you create an agent that produces consistent, structured, and actionable feedback. The framework's validation guarantees mean you can trust the shape of every response, making integration into CI/CD pipelines and developer tooling straightforward. Start with the agent in this tutorial, iterate on your schema and prompts based on real review quality, and you'll have a reliable code review assistant that scales with your team's needs.