Building a Code Review Agent with vLLM: Complete Guide

Building a Code Review Agent with vLLM: Complete Guide

Code reviews are one of the most valuable — yet time-consuming — practices in modern software development. With the rise of high-quality open-source large language models (LLMs), it's now possible to build a custom code review agent that runs on your own infrastructure, keeps your code private, and integrates directly into your workflow. In this guide, we'll walk through building a production-ready code review agent powered by vLLM, a fast and efficient inference engine for LLMs.

What Is vLLM?

vLLM is an open-source inference engine designed specifically for serving LLMs at high throughput and low latency. It was developed at UC Berkeley and has become the de facto standard for self-hosted LLM serving. Its key innovations include PagedAttention for efficient memory management, continuous batching for maximizing GPU utilization, and support for a wide range of popular open-source models.

Unlike managed API services, vLLM lets you run models like Llama 3, Qwen2.5-Coder, DeepSeek-Coder, or Mistral entirely on your own hardware. This is particularly important for code review, where you're dealing with proprietary source code that may have strict confidentiality requirements.

Why Build a Code Review Agent?

• Privacy and security: Your code never leaves your infrastructure.
• Cost efficiency: No per-token API charges; you pay only for your hardware.
• Customization: Fine-tune prompts and models to your team's coding standards.
• Integration: Connect directly to Git hooks, CI/CD pipelines, or pull request workflows.
• Consistency: Automated reviews catch common issues that human reviewers might miss under time pressure.

Prerequisites and Setup

Before we begin, you'll need a machine with a GPU. For code review tasks with models in the 7B–14B parameter range, a single NVIDIA GPU with 16GB+ of VRAM is sufficient. For larger models, you'll need correspondingly more memory.

Installing vLLM

Install vLLM using pip. We recommend using a fresh virtual environment or conda environment to avoid dependency conflicts:

# Create and activate a virtual environment
python -m venv vllm-env
source vllm-env/bin/activate

# Install vLLM
pip install vllm

# Install additional dependencies for our agent
pip install httpx pydantic tiktoken

Verify your installation by checking that vLLM can detect your GPU:

python -c "import vllm; print(vllm.__version__)"
nvidia-smi

Choosing a Model

For code review, you want a model that understands code semantics, can identify bugs, and communicates findings clearly. Good choices include:

• Qwen2.5-Coder-7B-Instruct — Excellent code understanding, fits on modest hardware.
• Qwen2.5-Coder-32B-Instruct — Top-tier code reasoning, requires more VRAM.
• DeepSeek-Coder-V2-Lite-Instruct — Strong multi-language support.
• Llama-3.1-8B-Instruct — General-purpose but capable with code.

For this tutorial, we'll use Qwen2.5-Coder-7B-Instruct as it offers an excellent balance of quality and resource requirements.

Starting the vLLM Server

vLLM provides an OpenAI-compatible API server, which means we can use the same client libraries and patterns we'd use with OpenAI's API. Start the server with the following command:

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-Coder-7B-Instruct \
  --port 8000 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90 \
  --tensor-parallel-size 1

Let's break down these flags:

• --model: The Hugging Face model identifier to load.
• --port: The port to serve the API on.
• --max-model-len: Maximum context length. Code reviews need room for both the diff and the response.
• --gpu-memory-utilization: Fraction of GPU memory to use. Leave some headroom for system processes.
• --tensor-parallel-size: Number of GPUs to split the model across. Use 1 for a single GPU.

Once the server is running, you can test it with a simple curl request:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-Coder-7B-Instruct",
    "messages": [
      {"role": "user", "content": "What does the Python function len() do?"}
    ],
    "max_tokens": 200
  }'

Building the Code Review Agent

Now let's build the actual agent. We'll create a Python module that takes a code diff, sends it to the vLLM server with a carefully crafted prompt, and returns structured review feedback.

Project Structure

code-review-agent/
├── agent.py          # Main agent logic
├── prompts.py        # Prompt templates
├── git_utils.py      # Git diff extraction
├── reviewer.py       # Review orchestration
├── config.py         # Configuration
└── main.py           # CLI entry point

Configuration

Start with a configuration module that centralizes all settings:

# config.py
from dataclasses import dataclass

@dataclass
class Config:
    vllm_base_url: str = "http://localhost:8000/v1"
    model_name: str = "Qwen/Qwen2.5-Coder-7B-Instruct"
    max_tokens: int = 4096
    temperature: float = 0.1
    top_p: float = 0.95
    # Maximum lines of diff to review in a single request
    max_diff_lines: int = 500

config = Config()

We use a low temperature (0.1) because code review requires focused, deterministic analysis rather than creative variation.

Prompt Engineering for Code Review

The quality of your agent's output depends heavily on the prompt. A good code review prompt should define the agent's role, specify what to look for, and request structured output.

# prompts.py

SYSTEM_PROMPT = """You are an expert code reviewer with deep knowledge of software engineering best practices, security, and performance optimization.

Your task is to review code changes (diffs) and provide actionable, specific feedback.

Focus on:
1. **Bugs and logic errors** - Incorrect conditions, off-by-one errors, null/None handling, race conditions
2. **Security vulnerabilities** - Injection attacks, hardcoded secrets, unsafe deserialization, path traversal
3. **Performance issues** - Unnecessary allocations, N+1 queries, inefficient algorithms, missing indexes
4. **Code quality** - Naming, readability, duplication, overly complex logic, missing error handling
5. **Best practices** - Framework conventions, design patterns, SOLID principles, DRY

Rules:
- Be specific: reference exact line numbers and code snippets
- Be constructive: suggest fixes, don't just point out problems
- Prioritize: focus on issues that matter, not trivial style nitpicks
- If the code is correct and well-written, say so clearly
- Use the output format specified by the user"""

def build_review_prompt(diff: str, file_context: str = "") -> str:
    prompt = f"""Review the following code diff and provide structured feedback.

"""
    if file_context:
        prompt += f"""## Surrounding Code Context
{file_context}
"""
    prompt += f"""## Code Diff to Review
diff
{diff}
## Output Format
Provide your review in the following format:

### Summary
[One or two sentence overview of the changes and overall quality]

### Issues Found
For each issue, use this structure:
- **Severity**: [Critical/Warning/Suggestion]
- **Location**: [File and line reference]
- **Description**: [What the issue is]
- **Suggestion**: [How to fix it, with code if applicable]

If no issues are found, write "No significant issues found."

### Positive Notes
[Mention anything done well, if applicable]
"""
    return prompt

Git Diff Extraction

Next, create utilities to extract diffs from Git. This allows the agent to work with real code changes:

# git_utils.py
import subprocess
from typing import Optional

def get_staged_diff() -> str:
    """Get the diff of staged changes (for pre-commit hook usage)."""
    result = subprocess.run(
        ["git", "diff", "--cached", "--unified=5"],
        capture_output=True,
        text=True,
        check=True
    )
    return result.stdout

def get_branch_diff(base_branch: str = "main") -> str:
    """Get the diff between current branch and a base branch."""
    result = subprocess.run(
        ["git", "diff", f"{base_branch}...HEAD", "--unified=5"],
        capture_output=True,
        text=True,
        check=True
    )
    return result.stdout

def get_file_diff(filepath: str, commit: Optional[str] = None) -> str:
    """Get the diff for a specific file, optionally at a specific commit."""
    if commit:
        cmd = ["git", "diff", f"{commit}~1", commit, "--", filepath]
    else:
        cmd = ["git", "diff", "HEAD", "--", filepath]
    result = subprocess.run(
        cmd, capture_output=True, text=True, check=True
    )
    return result.stdout

def get_file_content(filepath: str, commit: str = "HEAD") -> str:
    """Get the full content of a file at a given commit."""
    result = subprocess.run(
        ["git", "show", f"{commit}:{filepath}"],
        capture_output=True, text=True
    )
    return result.stdout if result.returncode == 0 else ""

The Core Agent

Now create the main agent that communicates with the vLLM server:

# agent.py
import httpx
import json
from config import config
from prompts import SYSTEM_PROMPT, build_review_prompt

class CodeReviewAgent:
    def __init__(self, base_url: str = None, model_name: str = None):
        self.base_url = base_url or config.vllm_base_url
        self.model_name = model_name or config.model_name
        self.client = httpx.Client(timeout=120.0)

    def review(self, diff: str, file_context: str = "") -> str:
        """Send a code diff to the LLM and get a review back."""
        if not diff.strip():
            return "No changes to review."

        user_prompt = build_review_prompt(diff, file_context)

        payload = {
            "model": self.model_name,
            "messages": [
                {"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": user_prompt}
            ],
            "max_tokens": config.max_tokens,
            "temperature": config.temperature,
            "top_p": config.top_p,
            "stream": False
        }

        response = self.client.post(
            f"{self.base_url}/chat/completions",
            json=payload
        )
        response.raise_for_status()

        data = response.json()
        return data["choices"][0]["message"]["content"]

    def review_streaming(self, diff: str, file_context: str = ""):
        """Stream the review response token by token."""
        user_prompt = build_review_prompt(diff, file_context)

        payload = {
            "model": self.model_name,
            "messages": [
                {"role": "system", "content": SYSTEM_PROMPT},
                {"role": "user", "content": user_prompt}
            ],
            "max_tokens": config.max_tokens,
            "temperature": config.temperature,
            "stream": True
        }

        with self.client.stream(
            "POST",
            f"{self.base_url}/chat/completions",
            json=payload
        ) as response:
            for line in response.iter_lines():
                if line.startswith("data: "):
                    data_str = line[6:]
                    if data_str == "[DONE]":
                        break
                    try:
                        chunk = json.loads(data_str)
                        delta = chunk["choices"][0]["delta"].get("content", "")
                        if delta:
                            yield delta
                    except json.JSONDecodeError:
                        continue

    def close(self):
        self.client.close()

Review Orchestration

Large diffs may exceed the model's context window or produce lower-quality reviews. The orchestrator splits diffs by file and processes each one separately:

# reviewer.py
import re
from typing import List, Dict
from agent import CodeReviewAgent
from config import config

class DiffParser:
    """Parse a unified diff into per-file segments."""

    @staticmethod
    def split_by_file(diff: str) -> List[Dict[str, str]]:
        files = []
        # Match diff headers like: diff --git a/file.py b/file.py
        pattern = r'(^diff --git .+$)'
        parts = re.split(pattern, diff, flags=re.MULTILINE)

        current_header = None
        for part in parts:
            if part.startswith("diff --git"):
                current_header = part
            elif current_header:
                filepath = DiffParser._extract_filepath(current_header)
                files.append({
                    "file": filepath,
                    "diff": current_header + "\n" + part
                })
                current_header = None
        return files

    @staticmethod
    def _extract_filepath(header: str) -> str:
        match = re.search(r'diff --git a/(.+?) b/', header)
        return match.group(1) if match else "unknown"

class CodeReviewer:
    def __init__(self):
        self.agent = CodeReviewAgent()

    def review_diff(self, diff: str) -> Dict[str, str]:
        """Review a full diff, splitting by file if necessary."""
        if not diff.strip():
            return {"_summary": "No changes to review."}

        files = DiffParser.split_by_file(diff)
        if not files:
            # Single file or unparseable, review as-is
            return {"_whole": self.agent.review(diff)}

        results = {}
        for file_info in files:
            file_diff = file_info["diff"]
            line_count = file_diff.count('\n')

            if line_count > config.max_diff_lines:
                results[file_info["file"]] = (
                    f"Skipped: diff too large ({line_count} lines). "
                    f"Please review manually or split the changes."
                )
                continue

            try:
                review = self.agent.review(file_diff)
                results[file_info["file"]] = review
            except Exception as e:
                results[file_info["file"]] = f"Review failed: {str(e)}"

        return results

    def close(self):
        self.agent.close()

CLI Entry Point

Finally, create a command-line interface so you can run the agent from your terminal:

# main.py
import sys
import argparse
from reviewer import CodeReviewer
from git_utils import get_staged_diff, get_branch_diff

def format_results(results: dict) -> str:
    output = []
    for filepath, review in results.items():
        if filepath.startswith("_"):
            output.append(review)
        else:
            output.append(f"\n{'='*60}")
            output.append(f"FILE: {filepath}")
            output.append(f"{'='*60}\n")
            output.append(review)
    return "\n".join(output)

def main():
    parser = argparse.ArgumentParser(description="AI Code Review Agent")
    parser.add_argument(
        "--mode",
        choices=["staged", "branch"],
        default="staged",
        help="Review staged changes or branch diff against main"
    )
    parser.add_argument(
        "--base-branch",
        default="main",
        help="Base branch for branch mode (default: main)"
    )
    parser.add_argument(
        "--stream",
        action="store_true",
        help="Stream output token by token"
    )
    args = parser.parse_args()

    if args.mode == "staged":
        diff = get_staged_diff()
    else:
        diff = get_branch_diff(args.base_branch)

    if not diff.strip():
        print("No changes to review.")
        sys.exit(0)

    reviewer = CodeReviewer()
    try:
        results = reviewer.review_diff(diff)
        print(format_results(results))
    finally:
        reviewer.close()

if __name__ == "__main__":
    main()

Running the Agent

With the vLLM server running, you can now review your staged changes:

# Stage some changes
git add src/auth.py

# Run the review agent
python main.py --mode staged

Or review an entire feature branch:

python main.py --mode branch --base-branch develop

Integrating with Git Hooks

To make the agent part of your development workflow, integrate it as a pre-commit hook. This runs an automatic review every time you try to commit changes:

#!/usr/bin/env bash
# .git/hooks/pre-commit

# Run the AI code review agent on staged changes
REVIEW_OUTPUT=$(python /path/to/code-review-agent/main.py --mode staged 2>&1)

# Print the review
echo "$REVIEW_OUTPUT"

# Check for critical issues (customize this logic)
if echo "$REVIEW_OUTPUT" | grep -q "Severity.*Critical"; then
    echo ""
    echo "⚠️  Critical issues detected by AI reviewer."
    echo "Review the feedback above. To commit anyway, use --no-verify"
    exit 1
fi

exit 0

Make the hook executable:

chmod +x .git/hooks/pre-commit

Integrating with CI/CD

For team-wide adoption, integrate the agent into your CI pipeline. Here's an example GitHub Actions workflow that posts review comments on pull requests:

# .github/workflows/ai-code-review.yml
name: AI Code Review

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  review:
    runs-on: self-hosted
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Start vLLM server
        run: |
          python -m vllm.entrypoints.openai.api_server \
            --model Qwen/Qwen2.5-Coder-7B-Instruct \
            --port 8000 \
            --max-model-len 16384 \
            --gpu-memory-utilization 0.90 &
          # Wait for server to be ready
          until curl -s http://localhost:8000/v1/models; do
            sleep 5
          done

      - name: Run code review
        run: |
          python main.py --mode branch --base-branch origin/main > review.md

      - name: Post review comment
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const review = fs.readFileSync('review.md', 'utf8');
            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: `## 🤖 AI Code Review\n\n${review}`
            });

Best Practices

1. Keep Diffs Focused

The agent performs best when reviewing focused, coherent changes. Very large diffs (hundreds of files) will produce shallow reviews. Encourage your team to make small, focused commits. The max_diff_lines configuration helps enforce this by skipping oversized diffs.

2. Provide Context

A diff alone sometimes lacks the context needed to evaluate whether code is correct. Consider passing surrounding code or related files. You can extend the get_file_content utility to fetch full file contents and include them in the prompt.

3. Use Structured Output

For programmatic processing (like the Git hook that checks for critical issues), consider requesting JSON output. You can modify the prompt to ask for a JSON array of issues:

JSON_OUTPUT_FORMAT = """Respond with ONLY a JSON object in this exact format:
{
  "summary": "Brief overview",
  "issues": [
    {
      "severity": "critical|warning|suggestion",
      "file": "path/to/file.py",
      "line": 42,
      "description": "What's wrong",
      "suggestion": "How to fix it"
    }
  ],
  "positive_notes": "What was done well"
}"""

Then parse the response with error handling:

import json

def parse_structured_review(response: str) -> dict:
    """Extract JSON from the LLM response, handling markdown fences."""
    # Remove markdown code fences if present
    cleaned = response.strip()
    if cleaned.startswith(""):
        lines = cleaned.split("\n")
        # Remove first and last lines (fences)
        lines = [l for l in lines if not l.strip().startswith("")]
        cleaned = "\n".join(lines)
    try:
        return json.loads(cleaned)
    except json.JSONDecodeError:
        return {"summary": response, "issues": [], "parse_error": True}

4. Cache and Batch Requests

If you're reviewing many files, vLLM's continuous batching handles concurrent requests efficiently. You can use asyncio with httpx.AsyncClient to send multiple file reviews in parallel:

import asyncio
import httpx

async def review_files_async(diffs: list, base_url: str, model: str) -> list:
    async with httpx.AsyncClient(timeout=120.0) as client:
        tasks = []
        for diff in diffs:
            payload = {
                "model": model,
                "messages": [
                    {"role": "system", "content": SYSTEM_PROMPT},
                    {"role": "user", "content": build_review_prompt(diff)}
                ],
                "max_tokens": 4096,
                "temperature": 0.1
            }
            tasks.append(client.post(f"{base_url}/chat/completions", json=payload))
        responses = await asyncio.gather(*tasks, return_exceptions=True)
        return [r.json()["choices"][0]["message"]["content"]
                if not isinstance(r, Exception) else f"Error: {r}"
                for r in responses]

5. Fine-Tune for Your Codebase

For even better results, consider fine-tuning a model on your team's past code reviews. Collect historical PR comments and review feedback, format them as instruction-response pairs, and fine-tune using LoRA or QLoRA. vLLM supports loading LoRA adapters at inference time with the --enable-lora flag.

6. Monitor and Iterate

Track the quality of reviews over time. Log which suggestions were accepted or rejected by developers. Use this feedback to refine your prompts or retrain your model. A simple logging mechanism can be added to the agent:

import logging
from datetime import datetime

logging.basicConfig(
    filename="code_review_agent.log",
    level=logging.INFO,
    format="%(asctime)s - %(levelname)s - %(message)s"
)

# In the review method:
logging.info(f"Reviewing diff of {len(diff)} chars, response: {len(review)} chars")

7. Set Appropriate Resource Limits

In production, protect your vLLM server from overload. Use rate limiting, set reasonable max_tokens limits, and monitor GPU memory usage. vLLM also supports quantized models (AWQ, GPTQ) that reduce memory requirements significantly with minimal quality loss:

python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-Coder-7B-Instruct-AWQ \
  --quantization awq \
  --port 8000 \
  --max-model-len 16384

Conclusion

Building a code review agent with vLLM gives you a powerful, private, and customizable tool for improving code quality across your team. By leveraging open-source models served through vLLM's high-performance inference engine, you get the benefits of AI-assisted review without sending proprietary code to third-party APIs. The architecture we've built — with diff extraction, per-file review orchestration, structured prompts, and Git/CI integration — provides a solid foundation that you can extend with fine-tuning, parallel processing, and custom review rules tailored to your organization's standards. Start with the default configuration, measure the quality of reviews on real pull requests, and iteratively refine your prompts and model selection to match your team's needs. With thoughtful implementation, an AI code review agent becomes a valuable second pair of eyes that catches issues early and reinforces best practices across every commit.