Building an Agent Evaluation Harness with LlamaIndex: Complete Guide

Agents powered by LLMs are increasingly being deployed in production environments to handle complex, multi-step tasks. But as agent behavior becomes more autonomous and unpredictable, traditional unit tests fall short. You need a systematic way to measure whether your agent is actually doing what it should. That's where an evaluation harness comes in. In this guide, we'll build a complete agent evaluation harness using LlamaIndex, covering everything from basic concepts to production-ready implementations.

What Is an Agent Evaluation Harness?

An agent evaluation harness is a structured framework that runs your agent against a predefined set of test cases and measures its performance using quantitative and qualitative metrics. Think of it as a test suite, but instead of asserting exact outputs, you evaluate reasoning quality, tool usage accuracy, response faithfulness, and task completion.

LlamaIndex provides a rich evaluation module through llama-index-core and llama-index-llms packages. It includes built-in evaluators for faithfulness, relevance, correctness, and more. By combining these with custom evaluators, you can build a harness tailored to your agent's specific use case.

Why It Matters

Agent evaluation is not optional if you care about reliability. Here's why building a dedicated harness matters:

Regression detection: When you change a prompt, swap a model, or add a tool, you need to know immediately if performance degraded.
Tool usage validation: Agents that call tools incorrectly can cause real damage. An evaluation harness catches these issues before deployment.
Cost and latency tracking: Evaluation runs give you data on token usage and response times across scenarios.
Confidence in deployment: A passing evaluation suite gives you the confidence to ship changes to production.
Comparative benchmarking: You can compare different models, prompts, or architectures on the same set of tasks.

Setting Up Your Environment

Before building the harness, install the required packages. We'll use LlamaIndex's core evaluation tools along with an OpenAI LLM for both the agent and the evaluator.

pip install llama-index-core llama-index-llms-openai llama-index-embeddings-openai
pip install python-dotenv pandas

Create a .env file with your API key:

OPENAI_API_KEY=sk-your-key-here

Now set up the basic imports and configuration:

import os
from dotenv import load_dotenv
from llama_index.core import Settings
from llama_index.llms.openai import OpenAI
from llama_index.embeddings.openai import OpenAIEmbedding

load_dotenv()

# Configure the global LLM used by evaluators
Settings.llm = OpenAI(model="gpt-4o", temperature=0)
Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small")

Building a Sample Agent to Evaluate

To demonstrate the harness, we need an agent. Let's build a simple research assistant agent that has access to a few tools. This agent will answer questions about documents and perform basic calculations.

from llama_index.core.agent import ReActAgent
from llama_index.core.tools import FunctionTool

def multiply_numbers(a: float, b: float) -> float:
    """Multiply two numbers together and return the result."""
    return a * b

def add_numbers(a: float, b: float) -> float:
    """Add two numbers together and return the result."""
    return a + b

multiply_tool = FunctionTool.from_defaults(fn=multiply_numbers)
add_tool = FunctionTool.from_defaults(fn=add_numbers)

agent = ReActAgent.from_tools(
    [multiply_tool, add_tool],
    llm=OpenAI(model="gpt-4o", temperature=0),
    verbose=True,
    system_prompt=(
        "You are a helpful math assistant. "
        "Always use the available tools for calculations. "
        "Show your reasoning step by step."
    ),
)

This agent uses the ReAct pattern, which reasons through a problem and then acts by calling tools. Now we need to evaluate whether it does this correctly.

Designing the Evaluation Dataset

The foundation of any evaluation harness is the dataset of test cases. Each test case should include a query, the expected behavior, and any reference answer or expected tool calls. Let's define a structured dataset.

from dataclasses import dataclass, field
from typing import List, Optional

@dataclass
class EvalTestCase:
    query: str
    expected_answer: Optional[str] = None
    expected_tools: List[str] = field(default_factory=list)
    reference_context: Optional[str] = None
    tags: List[str] = field(default_factory=list)

eval_dataset = [
    EvalTestCase(
        query="What is 15 multiplied by 23?",
        expected_answer="345",
        expected_tools=["multiply_numbers"],
        tags=["arithmetic", "multiplication"],
    ),
    EvalTestCase(
        query="Add 100 and 250, then tell me the result.",
        expected_answer="350",
        expected_tools=["add_numbers"],
        tags=["arithmetic", "addition"],
    ),
    EvalTestCase(
        query="What is the capital of France?",
        expected_answer="Paris",
        expected_tools=[],
        tags=["general_knowledge", "no_tool"],
    ),
    EvalTestCase(
        query="Multiply 7 by 8 and then add 10 to the result.",
        expected_answer="66",
        expected_tools=["multiply_numbers", "add_numbers"],
        tags=["arithmetic", "multi_step"],
    ),
]

Notice the last test case requires two tool calls in sequence. This tests the agent's ability to chain operations, which is a common failure point. The third test case checks that the agent does not call a tool when it doesn't need to, which is another common issue with over-eager agents.

Implementing the Evaluation Harness

Now we build the core harness. The harness runs each test case through the agent, captures the response and tool call metadata, and then runs evaluators against the results.

Capturing Agent Responses and Tool Calls

First, we need a function that runs the agent and captures detailed output, including which tools were called:

import time
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional

@dataclass
class AgentRunResult:
    query: str
    response: str
    tool_calls: List[str]
    latency_seconds: float
    token_usage: Dict[str, int]
    raw_output: Any

def run_agent_on_query(agent, query: str) -> AgentRunResult:
    start_time = time.time()
    
    # Capture tool calls by hooking into the agent's tool runner
    tool_calls_made = []
    original_tool_call = agent.tool_call
    
    def tracking_tool_call(*args, **kwargs):
        tool_name = kwargs.get("tool_name", args[0] if args else "unknown")
        tool_calls_made.append(tool_name)
        return original_tool_call(*args, **kwargs)
    
    agent.tool_call = tracking_tool_call
    
    try:
        response = agent.chat(query)
    finally:
        agent.tool_call = original_tool_call
    
    latency = time.time() - start_time
    
    # Extract token usage if available
    token_usage = {}
    if hasattr(response, "raw") and response.raw:
        usage = response.raw.get("usage", {})
        token_usage = {
            "prompt_tokens": usage.get("prompt_tokens", 0),
            "completion_tokens": usage.get("completion_tokens", 0),
            "total_tokens": usage.get("total_tokens", 0),
        }
    
    return AgentRunResult(
        query=query,
        response=str(response),
        tool_calls=tool_calls_made,
        latency_seconds=latency,
        token_usage=token_usage,
        raw_output=response,
    )

Building Evaluators

LlamaIndex provides several built-in evaluators. We'll use a combination of built-in and custom evaluators. Let's start with the built-in ones for answer correctness and faithfulness, then add custom evaluators for tool usage.

from llama_index.core.evaluation import (
    CorrectnessEvaluator,
    FaithfulnessEvaluator,
    RelevancyEvaluator,
)

# Initialize LlamaIndex built-in evaluators
correctness_evaluator = CorrectnessEvaluator(llm=Settings.llm)
faithfulness_evaluator = FaithfulnessEvaluator(llm=Settings.llm)
relevancy_evaluator = RelevancyEvaluator(llm=Settings.llm)

Now let's create a custom evaluator for tool usage accuracy. This checks whether the agent called the right tools and didn't call unnecessary ones:

@dataclass
class ToolUsageResult:
    passed: bool
    score: float
    reasoning: str
    missing_tools: List[str]
    unexpected_tools: List[str]

def evaluate_tool_usage(
    expected_tools: List[str],
    actual_tools: List[str]
) -> ToolUsageResult:
    expected_set = set(expected_tools)
    actual_set = set(actual_tools)
    
    missing = list(expected_set - actual_set)
    unexpected = list(actual_set - expected_set)
    
    if not expected_set and not actual_set:
        return ToolUsageResult(
            passed=True,
            score=1.0,
            reasoning="No tools expected and none called. Correct behavior.",
            missing_tools=[],
            unexpected_tools=[],
        )
    
    if not expected_set and actual_set:
        return ToolUsageResult(
            passed=False,
            score=0.0,
            reasoning=f"Agent called tools unnecessarily: {unexpected}",
            missing_tools=[],
            unexpected_tools=unexpected,
        )
    
    # Calculate F1-style score
    true_positives = len(expected_set & actual_set)
    precision = true_positives / len(actual_set) if actual_set else 0
    recall = true_positives / len(expected_set) if expected_set else 0
    
    if precision + recall == 0:
        f1 = 0.0
    else:
        f1 = 2 * (precision * recall) / (precision + recall)
    
    passed = len(missing) == 0 and len(unexpected) == 0
    
    reasoning_parts = []
    if missing:
        reasoning_parts.append(f"Missing expected tools: {missing}")
    if unexpected:
        reasoning_parts.append(f"Called unexpected tools: {unexpected}")
    if not reasoning_parts:
        reasoning_parts.append("All expected tools called correctly.")
    
    return ToolUsageResult(
        passed=passed,
        score=f1,
        reasoning="; ".join(reasoning_parts),
        missing_tools=missing,
        unexpected_tools=unexpected,
    )

Assembling the Harness

Now we bring everything together into a single harness class that runs all test cases and collects results:

import pandas as pd
from typing import List, Dict

class AgentEvalHarness:
    def __init__(self, agent, evaluators: Dict):
        self.agent = agent
        self.evaluators = evaluators
        self.results: List[Dict] = []
    
    async def run_single(self, test_case: EvalTestCase) -> Dict:
        # Run the agent
        run_result = run_agent_on_query(self.agent, test_case.query)
        
        # Run correctness evaluation if we have a reference answer
        correctness_result = None
        if test_case.expected_answer:
            correctness_result = await self.evaluators["correctness"].aevaluate(
                query=test_case.query,
                response=run_result.response,
                reference=test_case.expected_answer,
            )
        
        # Run tool usage evaluation
        tool_result = evaluate_tool_usage(
            test_case.expected_tools,
            run_result.tool_calls,
        )
        
        return {
            "query": test_case.query,
            "response": run_result.response,
            "expected_answer": test_case.expected_answer,
            "expected_tools": test_case.expected_tools,
            "actual_tools": run_result.tool_calls,
            "correctness_passed": correctness_result.passing if correctness_result else None,
            "correctness_score": correctness_result.score if correctness_result else None,
            "correctness_feedback": correctness_result.feedback if correctness_result else None,
            "tool_usage_passed": tool_result.passed,
            "tool_usage_score": tool_result.score,
            "tool_usage_reasoning": tool_result.reasoning,
            "latency_seconds": run_result.latency_seconds,
            "token_usage": run_result.token_usage,
            "tags": test_case.tags,
        }
    
    async def run_all(self, dataset: List[EvalTestCase]) -> pd.DataFrame:
        self.results = []
        for i, test_case in enumerate(dataset):
            print(f"Running test case {i+1}/{len(dataset)}: {test_case.query[:50]}...")
            result = await self.run_single(test_case)
            self.results.append(result)
        
        return pd.DataFrame(self.results)
    
    def summary(self) -> Dict:
        if not self.results:
            return {}
        
        df = pd.DataFrame(self.results)
        
        correctness_pass_rate = df["correctness_passed"].mean()
        tool_pass_rate = df["tool_usage_passed"].mean()
        avg_latency = df["latency_seconds"].mean()
        
        return {
            "total_cases": len(df),
            "correctness_pass_rate": f"{correctness_pass_rate:.1%}",
            "tool_usage_pass_rate": f"{tool_pass_rate:.1%}",
            "avg_latency_seconds": f"{avg_latency:.2f}s",
            "total_tokens": df["token_usage"].apply(
                lambda x: x.get("total_tokens", 0)
            ).sum(),
        }

Running the Evaluation

With the harness assembled, let's run it against our dataset:

import asyncio

evaluators = {
    "correctness": correctness_evaluator,
    "faithfulness": faithfulness_evaluator,
    "relevancy": relevancy_evaluator,
}

harness = AgentEvalHarness(agent=agent, evaluators=evaluators)

async def main():
    results_df = await harness.run_all(eval_dataset)
    
    print("\n" + "="*60)
    print("EVALUATION RESULTS")
    print("="*60)
    
    # Display individual results
    for _, row in results_df.iterrows():
        print(f"\nQuery: {row['query']}")
        print(f"Response: {row['response'][:100]}...")
        print(f"Expected tools: {row['expected_tools']}")
        print(f"Actual tools: {row['actual_tools']}")
        print(f"Tool usage passed: {row['tool_usage_passed']}")
        print(f"Correctness passed: {row['correctness_passed']}")
        print(f"Latency: {row['latency_seconds']:.2f}s")
    
    # Display summary
    print("\n" + "="*60)
    print("SUMMARY")
    print("="*60)
    summary = harness.summary()
    for key, value in summary.items():
        print(f"{key}: {value}")

asyncio.run(main())

Adding Advanced Evaluators

Beyond correctness and tool usage, you may want to evaluate other dimensions. Let's add a custom evaluator that checks whether the agent's reasoning is coherent and a latency-based evaluator for performance regression.

Reasoning Quality Evaluator

This evaluator uses an LLM to judge whether the agent's step-by-step reasoning makes sense:

from llama_index.core.base.llms.base import BaseLLM

class ReasoningQualityEvaluator:
    def __init__(self, llm: BaseLLM):
        self.llm = llm
        self.prompt_template = (
            "You are evaluating the reasoning quality of an AI agent.\n\n"
            "Query: {query}\n"
            "Agent Response: {response}\n"
            "Tools Called: {tools}\n\n"
            "Evaluate the reasoning on a scale of 1-5 where:\n"
            "1 = No reasoning, jumped to conclusion\n"
            "2 = Minimal reasoning, mostly correct\n"
            "3 = Adequate reasoning, some gaps\n"
            "4 = Clear reasoning, well-structured\n"
            "5 = Excellent reasoning, thorough and logical\n\n"
            "Respond in this format:\n"
            "SCORE: [number]\n"
            "FEEDBACK: [your explanation]"
        )
    
    async def evaluate(self, query: str, response: str, tools: List[str]) -> Dict:
        prompt = self.prompt_template.format(
            query=query,
            response=response,
            tools=", ".join(tools) if tools else "none",
        )
        
        result = await self.llm.acomplete(prompt)
        text = str(result)
        
        score = 0
        feedback = ""
        for line in text.split("\n"):
            if line.startswith("SCORE:"):
                try:
                    score = float(line.split(":")[1].strip())
                except ValueError:
                    score = 0
            elif line.startswith("FEEDBACK:"):
                feedback = line.split(":", 1)[1].strip()
        
        return {
            "score": score / 5.0,  # Normalize to 0-1
            "feedback": feedback,
            "passing": score >= 3,
        }

Integrating the Advanced Evaluator

Update the harness to include reasoning quality evaluation:

reasoning_evaluator = ReasoningQualityEvaluator(llm=Settings.llm)

# Add to the run_single method in AgentEvalHarness:
# reasoning_result = await reasoning_evaluator.evaluate(
#     query=test_case.query,
#     response=run_result.response,
#     tools=run_result.tool_calls,
# )
# Then add reasoning_result to the returned dictionary

Saving and Loading Evaluation Results

For regression testing, you need to persist results between runs. Here's how to save and compare results:

import json
from datetime import datetime

def save_results(harness: AgentEvalHarness, filepath: str):
    output = {
        "timestamp": datetime.now().isoformat(),
        "summary": harness.summary(),
        "results": harness.results,
    }
    with open(filepath, "w") as f:
        json.dump(output, f, indent=2, default=str)
    print(f"Results saved to {filepath}")

def load_results(filepath: str) -> Dict:
    with open(filepath, "r") as f:
        return json.load(f)

def compare_results(baseline_path: str, current_path: str) -> Dict:
    baseline = load_results(baseline_path)
    current = load_results(current_path)
    
    b_summary = baseline["summary"]
    c_summary = current["summary"]
    
    print("BASELINE vs CURRENT")
    print("-" * 40)
    for key in b_summary:
        if key in c_summary:
            print(f"{key}: {b_summary[key]} -> {c_summary[key]}")
    
    # Detect regressions
    regressions = []
    for key in ["correctness_pass_rate", "tool_usage_pass_rate"]:
        b_val = float(b_summary[key].strip("%")) / 100
        c_val = float(c_summary[key].strip("%")) / 100
        if c_val < b_val:
            regressions.append(f"{key} regressed from {b_summary[key]} to {c_summary[key]}")
    
    return {"regressions": regressions, "baseline": b_summary, "current": c_summary}

Best Practices

Building the harness is only half the battle. Here are best practices to ensure your evaluation is meaningful and maintainable:

Use a separate LLM for evaluation: The evaluator LLM should ideally be a stronger model than the agent LLM to avoid blind spots. If your agent uses GPT-4o-mini, evaluate with GPT-4o.
Keep your test dataset curated: Quality matters more than quantity. Fifty well-designed test cases are more valuable than five hundred generic ones.
Include edge cases deliberately: Test scenarios where the agent should refuse to act, where tools should not be called, and where multi-step reasoning is required.
Run evaluations on every change: Integrate the harness into your CI/CD pipeline so regressions are caught before merge.
Track metrics over time: Store results with timestamps and plot trends. A slow decline in performance can be harder to spot without historical data.
Set thresholds for pass/fail: Define what "good enough" means. A 90% tool usage pass rate might be acceptable for a prototype but not for production.
Don't over-rely on LLM-based evaluators: LLM judges can be biased or inconsistent. Combine them with deterministic checks wherever possible, like the tool usage evaluator we built.
Version your datasets: As your agent evolves, your test cases should too. Use versioning so you can compare apples to apples across runs.
Test for latency and cost: A correct answer that takes 30 seconds and costs $0.50 per query may not be viable. Track these metrics alongside quality.

Conclusion

Building an agent evaluation harness with LlamaIndex gives you a systematic, repeatable way to measure agent quality across multiple dimensions. By combining LlamaIndex's built-in evaluators for correctness and faithfulness with custom evaluators for tool usage and reasoning quality, you create a comprehensive safety net that catches regressions before they reach production. The harness we built in this guide is a starting point — as your agent grows in complexity, you can extend it with domain-specific evaluators, larger datasets, and deeper integration into your deployment pipeline. The key insight is that agent evaluation is not a one-time task but an ongoing practice that should evolve alongside your agent. Start small, run evaluations frequently, and let the data guide your decisions about when and how to improve your agent.