Introduction to Agent Evaluation Harnesses
Building AI agents is the easy part. Knowing whether they actually work — across edge cases, prompt variations, and tool combinations — is where most teams hit a wall. An agent evaluation harness is a structured system that runs your agents against a curated set of test cases, scores their outputs, and surfaces regressions before they reach production.
The OpenAI Agents SDK provides a lightweight, Pythonic framework for defining agents, tools, handoffs, and guardrails. While the SDK ships with tracing and observability, it does not include a built-in evaluation framework. That is by design — evaluation is highly domain-specific, and building your own harness gives you full control over scoring, fixtures, and CI integration.
In this guide, we will build a complete evaluation harness from scratch: defining test cases, running agents deterministically, scoring with both rule-based and LLM-as-judge evaluators, and wiring everything into a reproducible pipeline.
Why Evaluation Matters for Agents
Unlike traditional ML models, agents are non-deterministic by default. The same input can produce wildly different trajectories — different tool calls, different reasoning paths, different final answers. This makes evaluation both harder and more important.
Core Problems Evaluation Solves
- Regression detection: A prompt tweak that improves one scenario may silently break ten others.
- Tool usage correctness: Did the agent call the right tool with the right arguments? Did it hallucinate parameters?
- Trajectory quality: Did the agent take a sensible path, or did it loop, retry unnecessarily, or hand off inappropriately?
- Cost and latency budgeting: Evaluation surfaces how many tokens and round-trips each scenario consumes.
- Guardrail effectiveness: Did output validators catch unsafe or malformed responses?
Without a harness, teams rely on vibe checks and ad-hoc testing. That works for a prototype. It fails the moment you have multiple agents, handoffs, and real users.
Anatomy of an Evaluation Harness
A well-designed harness has five components:
- Fixtures: Input scenarios, expected outputs, and metadata.
- Runner: Executes the agent against each fixture and captures the full trace.
- Evaluators: Functions that score the agent's output and trajectory against expectations.
- Reporter: Aggregates scores into a pass/fail summary with per-case detail.
- CI integration: Runs the harness on every commit and fails the build on regressions.
We will build each of these against a concrete example: a customer support agent that can look up orders and issue refunds.
Setting Up the Project
Start by installing the Agents SDK and a few supporting dependencies:
pip install openai-agents pydantic rich pytest
Create the following project structure:
eval_harness/
├── agents/
│ └── support_agent.py
├── fixtures/
│ └── support_cases.py
├── evaluators/
│ ├── rule_based.py
│ └── llm_judge.py
├── harness/
│ ├── runner.py
│ ├── reporter.py
│ └── types.py
└── run_evals.py
Defining the Agent Under Test
First, define the agent you want to evaluate. We will keep it simple: a support agent with two tools — order lookup and refund issuance — plus a guardrail that blocks refund amounts over $500.
# agents/support_agent.py
from agents import Agent, function_tool, Runner
from pydantic import BaseModel
class Order(BaseModel):
order_id: str
status: str
total: float
# Simulated database
ORDERS = {
"ORD-1001": Order(order_id="ORD-1001", status="delivered", total=129.99),
"ORD-1002": Order(order_id="ORD-1002", status="shipped", total=49.00),
"ORD-1003": Order(order_id="ORD-1003", status="cancelled", total=899.00),
}
@function_tool
def lookup_order(order_id: str) -> Order:
"""Look up an order by its ID."""
if order_id not in ORDERS:
return Order(order_id=order_id, status="not_found", total=0.0)
return ORDERS[order_id]
@function_tool
def issue_refund(order_id: str, amount: float) -> str:
"""Issue a refund for a given order. Amount must not exceed the order total."""
order = ORDERS.get(order_id)
if not order:
return f"Order {order_id} not found."
if amount > order.total:
return f"Refund amount ${amount} exceeds order total ${order.total}."
return f"Refund of ${amount} issued for {order_id}."
support_agent = Agent(
name="SupportAgent",
instructions=(
"You are a customer support agent. Look up orders when asked. "
"Issue refunds only for delivered orders and only for amounts "
"up to $500. If a refund exceeds $500, explain that it requires "
"manager approval and do not call the refund tool."
),
tools=[lookup_order, issue_refund],
)
This agent has enough complexity to make evaluation meaningful: it must reason about order status, enforce a refund cap, and decide when to call a tool versus when to refuse.
Defining Test Fixtures
Fixtures are the heart of your harness. Each fixture describes an input, the expected behavior, and any metadata evaluators need. Use Pydantic to make fixtures type-safe and self-documenting.
# fixtures/support_cases.py
from pydantic import BaseModel
from typing import Optional, List
class Fixture(BaseModel):
id: str
description: str
user_message: str
expected_tool_calls: List[str] = []
forbidden_tool_calls: List[str] = []
expected_refund_amount: Optional[float] = None
must_mention: List[str] = []
must_not_mention: List[str] = []
FIXTURES: List[Fixture] = [
Fixture(
id="happy_path_refund",
description="Customer requests a refund for a delivered order under $500",
user_message="Please refund order ORD-1001. It arrived damaged.",
expected_tool_calls=["lookup_order", "issue_refund"],
expected_refund_amount=129.99,
must_mention=["refund"],
),
Fixture(
id="refund_exceeds_cap",
description="Customer requests refund over $500 — agent must refuse",
user_message="I want a full refund for ORD-1003.",
expected_tool_calls=["lookup_order"],
forbidden_tool_calls=["issue_refund"],
must_mention=["manager", "approval"],
),
Fixture(
id="unknown_order",
description="Customer references an order that does not exist",
user_message="Where is my order ORD-9999?",
expected_tool_calls=["lookup_order"],
forbidden_tool_calls=["issue_refund"],
must_not_mention=["refund"],
),
Fixture(
id="refund_shipped_order",
description="Customer requests refund for an order that has not been delivered",
user_message="Please refund ORD-1002, it hasn't arrived yet.",
expected_tool_calls=["lookup_order"],
forbidden_tool_calls=["issue_refund"],
must_mention=["shipped", "delivered"],
),
]
Notice how each fixture encodes both what the agent should do (expected tool calls) and what it must not do (forbidden tool calls, forbidden phrases). This dual structure catches both false positives and false negatives.
Capturing Agent Traces
To evaluate tool usage, we need to inspect what the agent actually did. The Agents SDK emits traces via its tracing system. We will use a custom processor to capture tool calls into a structured record.
# harness/types.py
from pydantic import BaseModel
from typing import Any, List, Optional
class ToolCallRecord(BaseModel):
name: str
arguments: dict
class AgentTrace(BaseModel):
fixture_id: str
final_output: str
tool_calls: List[ToolCallRecord]
raw_events: List[Any] = []
error: Optional[str] = None
class EvalResult(BaseModel):
fixture_id: str
passed: bool
scores: dict
details: List[str]
Now build the runner. The runner executes the agent, captures tool calls from the run result, and packages everything into an AgentTrace.
# harness/runner.py
import json
from agents import Runner
from .types import AgentTrace, ToolCallRecord
async def run_agent(agent, fixture) -> AgentTrace:
try:
result = await Runner.run(agent, input=fixture.user_message)
tool_calls = []
for item in result.new_items:
# The SDK exposes tool call items with a type discriminator
if getattr(item, "type", "") == "tool_call_item":
raw = getattr(item, "raw_item", None)
if raw is not None:
name = getattr(raw, "name", "unknown")
args_str = getattr(raw, "arguments", "{}")
try:
args = json.loads(args_str) if isinstance(args_str, str) else args_str
except json.JSONDecodeError:
args = {"_raw": args_str}
tool_calls.append(ToolCallRecord(name=name, arguments=args))
return AgentTrace(
fixture_id=fixture.id,
final_output=result.final_output,
tool_calls=tool_calls,
raw_events=[],
)
except Exception as e:
return AgentTrace(
fixture_id=fixture.id,
final_output="",
tool_calls=[],
error=str(e),
)
The exact attribute names for inspecting run items may vary between SDK versions. Always check the SDK's RunResult and item types in your installed version and adapt accordingly. The pattern — iterate new_items, filter by type, extract name and arguments — is stable across releases.
Writing Evaluators
Evaluators are pure functions that take a fixture and a trace and return a score. Start with deterministic, rule-based evaluators. They are fast, cheap, and unambiguous.
Rule-Based Evaluators
# evaluators/rule_based.py
from harness.types import AgentTrace, EvalResult
def eval_tool_calls(fixture, trace: AgentTrace) -> tuple[float, list[str]]:
details = []
called = {tc.name for tc in trace.tool_calls}
score = 1.0
for expected in fixture.expected_tool_calls:
if expected not in called:
score = 0.0
details.append(f"MISSING expected tool call: {expected}")
for forbidden in fixture.forbidden_tool_calls:
if forbidden in called:
score = 0.0
details.append(f"FORBIDDEN tool call made: {forbidden}")
if score == 1.0 and fixture.expected_tool_calls:
details.append("All expected tool calls present, no forbidden calls.")
return score, details
def eval_keyword_presence(fixture, trace: AgentTrace) -> tuple[float, list[str]]:
details = []
output = trace.final_output.lower()
score = 1.0
for keyword in fixture.must_mention:
if keyword.lower() not in output:
score = 0.0
details.append(f"MISSING required keyword: '{keyword}'")
for keyword in fixture.must_not_mention:
if keyword.lower() in output:
score = 0.0
details.append(f"FORBIDDEN keyword present: '{keyword}'")
return score, details
def eval_refund_amount(fixture, trace: AgentTrace) -> tuple[float, list[str]]:
if fixture.expected_refund_amount is None:
return 1.0, ["No refund amount expectation set."]
for tc in trace.tool_calls:
if tc.name == "issue_refund":
actual = tc.arguments.get("amount")
if actual == fixture.expected_refund_amount:
return 1.0, [f"Refund amount correct: ${actual}"]
return 0.0, [f"Refund amount mismatch: expected ${fixture.expected_refund_amount}, got ${actual}"]
return 0.0, ["Expected a refund but issue_refund was not called."]
def eval_no_error(fixture, trace: AgentTrace) -> tuple[float, list[str]]:
if trace.error:
return 0.0, [f"Agent raised an error: {trace.error}"]
return 1.0, ["No errors."]
Rule-based evaluators cover the mechanical aspects of agent behavior. They cannot judge nuance — whether a refusal was polite, whether an explanation was accurate, whether the agent understood the customer's intent. For that, use an LLM-as-judge evaluator.
LLM-as-Judge Evaluator
# evaluators/llm_judge.py
from agents import Agent, Runner
from harness.types import AgentTrace
JUDGE_INSTRUCTIONS = """You are an evaluation judge. You will receive a customer
message, the agent's response, and a description of what the correct behavior
should be. Score the agent's response from 0.0 to 1.0 on these criteria:
- Correctness: Did the agent do what the scenario requires?
- Clarity: Was the response clear and helpful?
- Safety: Did the agent avoid harmful or inappropriate actions?
Respond with ONLY a JSON object: {"score": float, "reasoning": string}
"""
judge_agent = Agent(
name="JudgeAgent",
instructions=JUDGE_INSTRUCTIONS,
output_type=dict,
)
async def eval_llm_judge(fixture, trace: AgentTrace) -> tuple[float, list[str]]:
prompt = (
f"Scenario description: {fixture.description}\n"
f"Customer message: {fixture.user_message}\n"
f"Agent response: {trace.final_output}\n"
f"Tool calls made: {[tc.name for tc in trace.tool_calls]}\n"
)
result = await Runner.run(judge_agent, input=prompt)
output = result.final_output
score = float(output.get("score", 0.0))
reasoning = output.get("reasoning", "No reasoning provided.")
return score, [f"LLM Judge: {reasoning}"]
The judge agent uses structured output to return a score and reasoning. Keep the judge's instructions narrow and specific — vague rubrics produce noisy scores. Always log the judge's reasoning so you can debug disagreements.
Assembling the Harness
Now wire the runner and evaluators together. The harness iterates fixtures, runs the agent, applies all evaluators, and aggregates results.
# harness/reporter.py
from rich.console import Console
from rich.table import Table
from harness.types import EvalResult
console = Console()
def print_report(results: list[EvalResult]) -> dict:
table = Table(title="Agent Evaluation Results")
table.add_column("Fixture", style="cyan")
table.add_column("Passed", style="bold")
table.add_column("Scores", style="magenta")
table.add_column("Details")
passed_count = 0
for r in results:
if r.passed:
passed_count += 1
scores_str = ", ".join(f"{k}={v:.2f}" for k, v in r.scores.items())
details_str = " | ".join(r.details[:3])
status = "[green]PASS[/green]" if r.passed else "[red]FAIL[/red]"
table.add_row(r.fixture_id, status, scores_str, details_str)
console.print(table)
total = len(results)
summary = {
"total": total,
"passed": passed_count,
"failed": total - passed_count,
"pass_rate": passed_count / total if total else 0.0,
}
console.print(
f"\n[bold]Summary:[/bold] {passed_count}/{total} passed "
f"({summary['pass_rate']:.0%})"
)
return summary
# run_evals.py
import asyncio
from agents.support_agent import support_agent
from fixtures.support_cases import FIXTURES
from harness.runner import run_agent
from harness.reporter import print_report
from harness.types import EvalResult
from evaluators.rule_based import (
eval_tool_calls,
eval_keyword_presence,
eval_refund_amount,
eval_no_error,
)
from evaluators.llm_judge import eval_llm_judge
EVALUATORS = [
("no_error", eval_no_error),
("tool_calls", eval_tool_calls),
("keywords", eval_keyword_presence),
("refund_amount", eval_refund_amount),
("llm_judge", eval_llm_judge),
]
async def evaluate_one(agent, fixture) -> EvalResult:
trace = await run_agent(agent, fixture)
scores = {}
details = []
all_passed = True
for name, evaluator in EVALUATORS:
score, eval_details = await evaluator(fixture, trace) \
if asyncio.iscoroutinefunction(evaluator) \
else evaluator(fixture, trace)
scores[name] = score
details.extend(eval_details)
if score < 1.0:
all_passed = False
return EvalResult(
fixture_id=fixture.id,
passed=all_passed,
scores=scores,
details=details,
)
async def main():
results = []
for fixture in FIXTURES:
result = await evaluate_one(support_agent, fixture)
results.append(result)
summary = print_report(results)
# Exit code for CI integration
if summary["failed"] > 0:
exit(1)
if __name__ == "__main__":
asyncio.run(main())
Run the harness:
python run_evals.py
You should see a table showing each fixture, whether it passed, individual evaluator scores, and a summary line. If any fixture fails, the script exits with code 1 — perfect for CI.
Best Practices
Design Fixtures for Failure, Not Just Success
The most valuable fixtures are the ones that test refusal, edge cases, and adversarial inputs. A harness with only happy-path fixtures will give you a false sense of confidence. Aim for at least 30% negative or boundary cases.
Pin Model Versions and Temperature
Non-determinism is the enemy of reproducible evaluation. Pin the model version (e.g., gpt-4o-2024-08-06) and set temperature to 0 for both the agent under test and the judge. If you need to measure variance, run each fixture N times and report the distribution rather than a single sample.
Separate Fast and Slow Evaluators
Rule-based evaluators are essentially free. LLM judges cost tokens and add latency. Run rule-based checks first and short-circuit: if a fixture fails a rule-based check, you may skip the judge call. This keeps your harness fast enough to run on every commit.
Version Your Fixtures
Treat fixtures like test code. Store them in version control, review changes in pull requests, and tag fixture sets to agent versions. When you ship a new agent prompt, you should know exactly which fixture set it was validated against.
Watch for Judge Bias
LLM judges have known biases: they prefer longer responses, agree with the agent more than they should, and are sensitive to prompt formatting. Mitigate this by using a different model family for the judge than for the agent, keeping rubrics explicit, and periodically auditing judge decisions by hand.
Integrate with pytest for Familiarity
If your team already uses pytest, you can wrap the harness as a pytest suite. This gives you parallel execution, fixtures, markers, and familiar reporting for free.
# test_agent_evals.py
import pytest
from agents.support_agent import support_agent
from fixtures.support_cases import FIXTURES
from run_evals import evaluate_one
@pytest.mark.asyncio
@pytest.mark.parametrize("fixture", FIXTURES, ids=[f.id for f in FIXTURES])
async def test_agent_fixture(fixture):
result = await evaluate_one(support_agent, fixture)
assert result.passed, f"Fixture {result.fixture_id} failed: {result.details}"
Track Scores Over Time
A single evaluation run tells you whether the agent passes today. Tracking scores across commits tells you whether the agent is improving or degrading. Store results as JSON, log them to your observability platform, and chart pass rate and per-evaluator scores over time. A slow drift downward is often more dangerous than a sudden break.
Conclusion
An evaluation harness transforms agent development from guesswork into engineering. By codifying expected behaviors as fixtures, scoring both mechanics and nuance with complementary evaluators, and running the whole pipeline in CI, you get fast, reliable feedback on every change. The OpenAI Agents SDK gives you the building blocks — agents, tools, tracing, structured outputs — and the harness pattern shown here wraps them in a repeatable evaluation workflow. Start with a small fixture set covering your most important scenarios, add evaluators incrementally as you discover failure modes, and treat the harness itself as a first-class part of your codebase. The agents you ship are only as trustworthy as the evaluation that backs them.