Building an Agent Evaluation Harness with vLLM: Complete Guide

As AI agents become more capable and are deployed in production environments, the need to rigorously evaluate their behavior grows. An agent evaluation harness is a structured framework that runs agents against a suite of tasks, captures their outputs, and scores them against expected outcomes. When paired with vLLM—a high-throughput, memory-efficient inference engine—you get a fast, scalable, and cost-effective way to benchmark agent performance locally or on your own infrastructure.

This guide walks through what an agent evaluation harness is, why vLLM is an excellent backend for one, and how to build a complete, working harness from scratch.

What Is an Agent Evaluation Harness?

An agent evaluation harness is a system that automates the process of testing AI agents. It typically consists of several components:

Task definitions — A set of prompts, environments, or scenarios the agent must navigate.
Agent runtime — The logic that drives the agent, including tool use, planning, and memory.
LLM backend — The inference engine that powers the agent's reasoning.
Scorer / evaluator — A mechanism that compares agent outputs to ground truth or rubrics.
Reporting layer — Aggregates results into metrics like pass rate, latency, and token usage.

Popular examples include OpenAI's evals framework, EleutherAI's lm-evaluation-harness, and SWE-bench. However, most of these focus on single-turn model evaluation. An agent harness must handle multi-turn interactions, tool calls, and stateful environments.

Why Build Your Own?

Off-the-shelf harnesses often make assumptions about the model provider, tool interface, or task format. Building your own gives you full control over:

The agent loop and tool definitions
The scoring criteria and rubrics
The inference backend and its performance characteristics
The data format and reporting pipeline

Why vLLM?

vLLM is an open-source inference engine that provides:

PagedAttention for efficient KV-cache management, enabling high throughput.
OpenAI-compatible API server, so agents can use standard client libraries.
Batched inference, which is critical when running hundreds of evaluation episodes in parallel.
Local deployment, meaning no per-token API costs and full data privacy.

For evaluation, these properties translate directly into faster iteration cycles and lower costs. A benchmark suite that would cost hundreds of dollars on a hosted API can run for free on a single GPU with vLLM.

Architecture Overview

Our harness will have the following structure:

A task loader that reads JSON task definitions.
An agent that uses a ReAct-style loop (reason → act → observe).
A vLLM client that communicates with a locally running vLLM server.
A scorer that uses both rule-based checks and an LLM-as-judge approach.
A runner that orchestrates everything and writes results to disk.

Prerequisites and Setup

You will need a machine with a CUDA-capable GPU (at least 16GB VRAM for a 7B model) and Python 3.10+. Install the required packages:

pip install vllm openai pydantic rich

Start the vLLM server in a separate terminal. We will use Qwen2.5-7B-Instruct as our agent model, but any instruct-tuned model works:

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --port 8000 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.9

Wait until you see the message indicating the server is ready. You can verify it is running with:

curl http://localhost:8000/v1/models

Defining Tasks

Tasks are the heart of any evaluation harness. Each task specifies a prompt, a set of available tools, and expected outcomes. We will define tasks in JSON for simplicity.

Create a file called tasks.json:

[
  {
    "id": "math_001",
    "category": "math",
    "prompt": "What is 17 multiplied by 23? Use the calculator tool.",
    "tools": ["calculator"],
    "expected_answer": "391",
    "max_steps": 5
  },
  {
    "id": "search_001",
    "category": "retrieval",
    "prompt": "What is the capital of Australia? Use the search tool.",
    "tools": ["search"],
    "expected_answer": "Canberra",
    "max_steps": 5
  },
  {
    "id": "multi_001",
    "category": "multi_step",
    "prompt": "Calculate the sum of the first 5 prime numbers, then multiply by 3.",
    "tools": ["calculator"],
    "expected_answer": "58",
    "max_steps": 8
  }
]

Each task has a unique ID, a category for grouping results, the tools the agent is allowed to use, the expected final answer, and a maximum number of reasoning steps to prevent infinite loops.

Building the Tool Layer

Agents need tools to act on the world. We will implement a small tool registry with two tools: a calculator and a mock search engine.

Create a file called tools.py:

import json
import re

TOOL_REGISTRY = {}

def register_tool(name):
    def decorator(func):
        TOOL_REGISTRY[name] = func
        return func
    return decorator

@register_tool("calculator")
def calculator(expression: str) -> str:
    """Evaluate a mathematical expression safely."""
    # Only allow numbers and basic operators
    if not re.match(r'^[\d\s\+\-\*\/\(\)\.]+$', expression):
        return "Error: invalid characters in expression"
    try:
        result = eval(expression, {"__builtins__": {}}, {})
        return str(result)
    except Exception as e:
        return f"Error: {e}"

@register_tool("search")
def search(query: str) -> str:
    """Mock search engine with a small knowledge base."""
    knowledge = {
        "capital of australia": "The capital of Australia is Canberra.",
        "capital of france": "The capital of France is Paris.",
        "population of japan": "Japan has a population of approximately 125 million.",
    }
    q = query.lower().strip()
    for key, value in knowledge.items():
        if key in q or q in key:
            return value
    return f"No results found for: {query}"

def execute_tool(name: str, args: dict) -> str:
    if name not in TOOL_REGISTRY:
        return f"Error: unknown tool '{name}'"
    try:
        return TOOL_REGISTRY[name](**args)
    except Exception as e:
        return f"Error executing tool {name}: {e}"

The registry pattern makes it easy to add new tools. Each tool takes keyword arguments and returns a string result, which keeps the interface uniform.

Building the Agent

Our agent uses a ReAct-style loop. At each step, it receives the conversation history and produces either a final answer or a tool call. We parse tool calls from the model output using a simple XML-like format.

Create a file called agent.py:

import json
from openai import OpenAI
from tools import execute_tool, TOOL_REGISTRY

SYSTEM_PROMPT = """You are a helpful assistant that solves tasks step by step.
You have access to tools. To use a tool, output exactly: