Building an Agent Evaluation Harness with llama.cpp: Complete Guide

Introduction to Agent Evaluation Harnesses

An agent evaluation harness is a systematic framework for measuring how well an autonomous LLM agent performs across a suite of tasks. Unlike simple benchmark scripts that check whether a model produces a specific string, an evaluation harness exercises the full agent loop: tool calls, multi-step reasoning, error recovery, and final answer synthesis. With llama.cpp becoming a popular choice for running quantized models locally, building an evaluation harness around it gives developers a reproducible, offline, and cost-effective way to compare agent architectures, prompts, and model sizes.

This guide walks through designing and implementing a complete evaluation harness from scratch. We will cover the architecture, the integration with llama.cpp's server, scoring strategies, and best practices for getting trustworthy results.

Why Build an Evaluation Harness?

When you ship an agent into production, you need confidence that it will behave correctly on unseen inputs. A harness gives you several concrete advantages:

Reproducibility: Every run uses the same task definitions, tools, and scoring logic, so changes in prompts or model weights can be attributed to the change itself rather than noise.
Regression detection: Continuous evaluation catches subtle degradations when you swap models or update tool schemas.
Cost control: Running locally with llama.cpp eliminates per-token API charges during development.
Privacy: Sensitive test data never leaves your machine.
Iteration speed: You can test dozens of prompt variants in minutes without rate limits.

Understanding the Components

A robust harness has four logical layers. Each layer should be isolated so you can swap implementations without rewriting the whole system.

1. The Model Server

llama.cpp ships an OpenAI-compatible HTTP server (llama-server). This is the easiest integration point because your harness can use the same client code you would use against a hosted API. Start it with a quantized model:

./llama-server \
  -m models/Meta-Llama-3-8B-Instruct.Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080 \
  -c 8192 -ngl 33 \
  --jinja

The --jinja flag enables the model's chat template, which is essential for tool-calling agents. The -ngl flag offloads layers to the GPU for faster inference, which matters because evaluation runs can involve thousands of completions.

2. The Agent Runtime

The agent runtime wraps the model server and implements the reasoning loop. It sends the prompt, parses tool calls, executes them, and feeds results back. Keeping this layer generic lets you evaluate different agent strategies (ReAct, function-calling, plan-and-execute) under the same harness.

3. The Task Suite

Tasks are declarative specifications of what the agent must accomplish. Each task defines the initial prompt, available tools, the maximum number of steps, and a scoring function. Storing tasks as YAML or JSON keeps them versionable and reviewable.

4. The Scorer

The scorer converts an agent's trajectory into a numeric score. Scoring can be exact-match, programmatic (running assertions against tool call arguments), or LLM-as-judge (using a stronger model to grade the trajectory).

Designing the Task Format

Consistency in task definition is what makes a harness trustworthy. Below is a minimal schema that captures everything a scorer needs.

# tasks/math_calc.yaml
id: math_calc_001
description: "Compute the area of a circle given radius via a calculator tool."
prompt: |
  What is the area of a circle with radius 7? Use the calculator tool.
  Round the result to two decimal places.
max_steps: 5
tools:
  - name: calculator
    description: "Evaluate a math expression."
    parameters:
      type: object
      properties:
        expression:
          type: string
      required: [expression]
scorer:
  type: tool_call_match
  expected_tool: calculator
  expected_expression_regex: "3\\.14.*7.*7|pi.*7.*7|7\\*\\*2.*pi"
  expected_final_answer_regex: "153\\.93|153\\.94"

Notice that the scorer combines two checks: that the agent called the right tool with a sensible expression, and that the final answer matches the expected value. This dual check prevents false positives where an agent produces the correct number through hallucination rather than computation.

Implementing the Agent Runtime

Below is a Python implementation of a minimal ReAct-style agent that talks to the llama.cpp server. It uses the OpenAI Python client for compatibility.

import json
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="not-needed")

def run_agent(prompt, tools, max_steps=5):
    messages = [{"role": "user", "content": prompt}]
    trajectory = {"steps": [], "final_answer": None}

    for step in range(max_steps):
        response = client.chat.completions.create(
            model="local",
            messages=messages,
            tools=tools,
            tool_choice="auto",
            temperature=0.0,
        )
        msg = response.choices[0].message

        if msg.tool_calls:
            messages.append(msg)
            for call in msg.tool_calls:
                args = json.loads(call.function.arguments)
                result = execute_tool(call.function.name, args)
                trajectory["steps"].append({
                    "tool": call.function.name,
                    "args": args,
                    "result": result,
                })
                messages.append({
                    "role": "tool",
                    "tool_call_id": call.id,
                    "content": json.dumps(result),
                })
        else:
            trajectory["final_answer"] = msg.content
            break

    return trajectory

def execute_tool(name, args):
    if name == "calculator":
        # Safe-ish eval for demo only; use a real parser in production.
        expr = args["expression"].replace("^", "**")
        try:
            return {"value": eval(expr, {"__builtins__": {}}, {})}
        except Exception as e:
            return {"error": str(e)}
    return {"error": f"unknown tool {name}"}

Setting temperature=0.0 is deliberate for evaluation. While it does not guarantee identical outputs across runs due to floating-point non-determinism in GPU inference, it minimizes variance and makes scores more comparable.

Building the Scorer

The scorer inspects the trajectory and returns a score between 0 and 1, plus a reason string for debugging. Here is a scorer that handles the schema defined earlier.

import re

def score_task(trajectory, scorer_spec):
    if scorer_spec["type"] == "tool_call_match":
        return score_tool_call_match(trajectory, scorer_spec)
    if scorer_spec["type"] == "exact_match":
        return score_exact_match(trajectory, scorer_spec)
    return {"score": 0.0, "reason": f"unknown scorer {scorer_spec['type']}"}

def score_tool_call_match(traj, spec):
    reasons = []
    score = 0.0

    expected_tool = spec.get("expected_tool")
    tool_calls = [s for s in traj["steps"] if s.get("tool") == expected_tool]
    if not tool_calls:
        return {"score": 0.0, "reason": f"never called {expected_tool}"}

    score += 0.4
    reasons.append("called expected tool")

    expr_re = spec.get("expected_expression_regex")
    if expr_re:
        matched = any(
            re.search(expr_re, json.dumps(s["args"]))
            for s in tool_calls
        )
        if matched:
            score += 0.3
            reasons.append("expression matched expected pattern")
        else:
            reasons.append("expression did not match")

    ans_re = spec.get("expected_final_answer_regex")
    if ans_re and traj["final_answer"]:
        if re.search(ans_re, traj["final_answer"]):
            score += 0.3
            reasons.append("final answer matched")
        else:
            reasons.append("final answer did not match")

    return {"score": round(score, 2), "reason": "; ".join(reasons)}

def score_exact_match(traj, spec):
    target = spec["expected"]
    got = (traj["final_answer"] or "").strip()
    ok = target.strip().lower() in got.lower()
    return {"score": 1.0 if ok else 0.0, "reason": "exact match" if ok else "no match"}

Splitting the score into weighted components is more informative than a binary pass/fail. You can see whether an agent fails because it never calls the tool, calls it incorrectly, or produces the wrong final synthesis. This granularity is what makes a harness useful for debugging.

Assembling the Harness Runner

The runner ties everything together: it loads tasks, runs the agent, scores results, and writes a report.

import yaml
import json
import time
from pathlib import Path

def load_tasks(task_dir):
    tasks = []
    for path in sorted(Path(task_dir).glob("*.yaml")):
        with open(path) as f:
            tasks.append(yaml.safe_load(f))
    return tasks

def run_harness(task_dir, output_path="results.json"):
    tasks = load_tasks(task_dir)
    results = []

    for task in tasks:
        start = time.time()
        try:
            traj = run_agent(
                task["prompt"],
                task["tools"],
                max_steps=task.get("max_steps", 5),
            )
            verdict = score_task(traj, task["scorer"])
        except Exception as e:
            traj = {"steps": [], "final_answer": None}
            verdict = {"score": 0.0, "reason": f"exception: {e}"}

        results.append({
            "id": task["id"],
            "score": verdict["score"],
            "reason": verdict["reason"],
            "elapsed_s": round(time.time() - start, 2),
            "steps_taken": len(traj["steps"]),
        })
        print(f"{task['id']}: {verdict['score']} - {verdict['reason']}")

    summary = {
        "total": len(results),
        "mean_score": round(sum(r["score"] for r in results) / max(len(results), 1), 3),
        "results": results,
    }
    with open(output_path, "w") as f:
        json.dump(summary, f, indent=2)
    return summary

if __name__ == "__main__":
    run_harness("tasks/")

Running this produces a JSON report with per-task scores and an aggregate mean. You can diff reports between runs to see whether a prompt change improved or hurt performance.

Adding LLM-as-Judge Scoring

Some tasks are too open-ended for regex matching. For those, use a stronger model as a judge. The trick is to use a different model instance (or a different endpoint) so the judge is not grading its own work.

JUDGE_PROMPT = """You are grading an AI agent's performance on a task.

Task prompt:
{prompt}

Agent trajectory (tool calls and results):
{trajectory}

Final answer:
{answer}

Grading criteria:
{criteria}

Respond with a JSON object: {{"score": float 0-1, "reason": "short explanation"}}
"""

def llm_judge(task, traj, judge_client):
    criteria = task["scorer"].get("criteria", "Did the agent complete the task correctly?")
    prompt = JUDGE_PROMPT.format(
        prompt=task["prompt"],
        trajectory=json.dumps(traj["steps"], indent=2),
        answer=traj["final_answer"],
        criteria=criteria,
    )
    resp = judge_client.chat.completions.create(
        model="judge",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.0,
        response_format={"type": "json_object"},
    )
    try:
        return json.loads(resp.choices[0].message.content)
    except json.JSONDecodeError:
        return {"score": 0.0, "reason": "judge returned invalid JSON"}

When using an LLM judge, always log the judge's reasoning. If the judge gives a suspicious score, the reasoning lets you audit whether the judge itself misunderstood the task.

Best Practices

Pin your model and quantization. Record the exact GGUF file and quantization level in every report. A Q4 and a Q8 of the same base model can produce noticeably different agent behavior.
Run multiple seeds. Even at temperature 0, GPU inference has minor nondeterminism. Run each task 3-5 times and report the mean and standard deviation.
Cap the step budget tightly. Generous step limits mask failures by letting agents brute-force solutions. Tight budgets reveal whether an agent is efficient.
Include negative tests. Add tasks where the correct behavior is to refuse or ask for clarification. Agents that always try to answer produce false positives on these.
Log full trajectories. Scores alone are not enough. Save the complete message history so you can replay and debug failures.
Isolate the judge. Never let the judge model be the same running instance as the agent under test. Use a separate port or a separate model file.
Version your tasks. Treat the task suite like test code. Use git, write commit messages, and review changes. A silently modified task invalidates historical comparisons.
Watch for context length drift. Long trajectories can exceed the context window mid-run. Log token counts per step so you can detect truncation.

Extending the Harness

Once the core is stable, you can extend it in several directions. Add parallel execution with a thread pool to speed up large suites, since the llama.cpp server can queue requests. Add a web dashboard that reads the results JSON and plots score trends over commits. Integrate the harness into CI so every pull request that touches agent code or prompts triggers a full evaluation run.

For multi-agent systems, generalize the task schema to declare multiple agent roles and a shared scratchpad. The scorer then evaluates the collective outcome rather than a single trajectory. This requires careful prompt design to prevent agents from talking past each other, but the harness structure remains the same.

Conclusion

Building an agent evaluation harness around llama.cpp gives you a private, reproducible, and inexpensive way to measure agent quality. By separating the model server, agent runtime, task definitions, and scorer into distinct layers, you create a system that can evolve as your agents grow more complex. Start with a small suite of well-specified tasks, score them with a mix of programmatic and LLM-judge methods, and treat the results as a first-class artifact alongside your code. The discipline of writing tasks and scorers forces you to articulate what correct agent behavior actually means, and that clarity pays off every time you ship a change.