Building an Agent Evaluation Harness with Claude Code: Complete Guide

Building an Agent Evaluation Harness with Claude Code: Complete Guide

As AI agents become more capable and are deployed in production environments, the need to systematically measure their performance grows. An agent evaluation harness is a structured framework that runs your agent against a curated set of tasks, captures its outputs, and scores them against expected outcomes. This guide walks you through building one using Claude Code, Anthropic's command-line coding agent, as both the system under test and a tool for generating evaluation infrastructure.

What Is an Agent Evaluation Harness?

An evaluation harness is a software system that automates the process of testing an autonomous agent. Unlike traditional unit tests, which check deterministic function outputs, an agent harness must account for variability: agents may take different paths to reach the same goal, produce natural language responses, and interact with external tools or environments.

A typical harness consists of four components:

• Task suite — a collection of test cases, each defining an input prompt, an environment setup, and success criteria.
• Runner — orchestrates execution, invoking the agent for each task with controlled inputs and timeouts.
• Environment — a sandbox (filesystem, container, or mock service) the agent operates within during evaluation.
• Scorer — evaluates the agent's outputs and side effects against the success criteria, producing quantitative metrics.

Claude Code is particularly well-suited as the agent under evaluation because it operates in a real filesystem, can execute shell commands, edit files, and reason multi-step — all behaviors you want to measure rigorously.

Why Evaluation Harnesses Matter

Without a harness, agent development relies on ad-hoc manual testing. You run a few prompts, eyeball the results, and ship. This approach breaks down quickly as capabilities grow and prompts, tools, or models change. A harness gives you:

• Reproducibility — the same task suite run against the same agent version produces comparable results.
• Regression detection — when you change a system prompt or swap a tool, you immediately see which tasks degrade.
• Capability baselines — quantitative scores let you compare models, configurations, or prompt variants.
• Targeted improvement — failing tasks become concrete work items rather than vague concerns.

In agentic systems, small prompt changes can cascade into large behavioral shifts. A harness is the only practical way to catch these before deployment.

Designing the Task Suite

The task suite is the heart of your harness. Each task should be self-contained, deterministic in setup, and have unambiguous success criteria. Let's define a task schema using TypeScript:

// src/types.ts
export interface AgentTask {
  id: string;
  description: string;
  prompt: string;
  setup: EnvironmentSetup;
  successCriteria: SuccessCriterion[];
  timeoutMs: number;
  maxTurns: number;
}

export interface EnvironmentSetup {
  files: Record<string, string>;  // path -> contents
  commands?: string[];                // shell commands to run before agent
}

export interface SuccessCriterion {
  type: 'file_exists' | 'file_contains' | 'command_succeeds' | 'llm_judge' | 'no_errors';
  description: string;
  params: Record<string, unknown>;
}

export interface TaskResult {
  taskId: string;
  passed: boolean;
  criteriaResults: CriterionResult[];
  agentLog: string;
  durationMs: number;
}

export interface CriterionResult {
  description: string;
  passed: boolean;
  detail: string;
}

Now let's create a sample task file. This task tests whether Claude Code can fix a broken TypeScript function:

// tasks/fix-typescript-error.json
{
  "id": "fix-ts-error-001",
  "description": "Agent must fix a TypeScript compilation error in a utility module",
  "prompt": "The file src/utils.ts has a TypeScript error. Fix it so that npm run build succeeds.",
  "setup": {
    "files": {
      "package.json": "{\"name\":\"test-proj\",\"scripts\":{\"build\":\"tsc\"},\"devDependencies\":{\"typescript\":\"^5.0.0\"}}",
      "tsconfig.json": "{\"compilerOptions\":{\"strict\":true,\"outDir\":\"dist\"},\"include\":[\"src\"]}",
      "src/utils.ts": "export function add(a: number, b: string): number {\n  return a + b;\n}\n"
    },
    "commands": ["npm install"]
  },
  "successCriteria": [
    {
      "type": "command_succeeds",
      "description": "Build passes after fix",
      "params": { "command": "npm run build" }
    },
    {
      "type": "file_contains",
      "description": "Function signature uses two numbers",
      "params": { "path": "src/utils.ts", "pattern": "add\\(a:\\s*number,\\s*b:\\s*number\\)" }
    }
  ],
  "timeoutMs": 120000,
  "maxTurns": 15
}

Building the Runner

The runner sets up an isolated environment for each task, invokes Claude Code, and collects results. We use temporary directories for isolation and the Claude Code CLI in non-interactive (print) mode:

// src/runner.ts
import { execSync, exec } from 'child_process';
import * as fs from 'fs/promises';
import * as path from 'path';
import { AgentTask, TaskResult, CriterionResult } from './types';
import { Scorer } from './scorer';

export class HarnessRunner {
  constructor(private scorer: Scorer) {}

  async runTask(task: AgentTask): Promise<TaskResult> {
    const workDir = await this.createEnvironment(task);
    const startTime = Date.now();

    let agentLog = '';
    let timedOut = false;

    try {
      agentLog = await this.invokeAgent(task, workDir);
    } catch (err) {
      timedOut = true;
      agentLog = `Agent execution failed: ${(err as Error).message}`;
    }

    const criteriaResults = await this.scorer.evaluate(task, workDir);
    const durationMs = Date.now() - startTime;

    return {
      taskId: task.id,
      passed: criteriaResults.every(c => c.passed) && !timedOut,
      criteriaResults,
      agentLog,
      durationMs,
    };
  }

  private async createEnvironment(task: AgentTask): Promise<string> {
    const workDir = path.join('/tmp/agent-eval', task.id + '-' + Date.now());
    await fs.mkdir(workDir, { recursive: true });

    for (const [filePath, contents] of Object.entries(task.setup.files)) {
      const fullPath = path.join(workDir, filePath);
      await fs.mkdir(path.dirname(fullPath), { recursive: true });
      await fs.writeFile(fullPath, contents);
    }

    for (const cmd of task.setup.commands ?? []) {
      execSync(cmd, { cwd: workDir, stdio: 'pipe', timeout: 60000 });
    }

    return workDir;
  }

  private invokeAgent(task: AgentTask, workDir: string): Promise<string> {
    return new Promise((resolve, reject) => {
      const cmd = `claude -p "${task.prompt.replace(/"/g, '\\"')}" --max-turns ${task.maxTurns} --output-format json`;

      exec(cmd, {
        cwd: workDir,
        timeout: task.timeoutMs,
        maxBuffer: 10 * 1024 * 1024,
      }, (error, stdout, stderr) => {
        if (error && error.killed) {
          reject(new Error('Agent timed out'));
          return;
        }
        resolve(stdout + (stderr ? '\n--- STDERR ---\n' + stderr : ''));
      });
    });
  }
}

Key design choices here: each task gets a fresh temporary directory so there is no cross-contamination. The agent runs with --max-turns to prevent runaway loops, and a hard timeout prevents hangs. The --output-format json flag gives us structured output we can parse later for deeper analysis.

Implementing the Scorer

The scorer checks each success criterion against the post-execution environment. Some criteria are deterministic (file existence, command exit codes), while others require an LLM judge for subjective quality assessment:

// src/scorer.ts
import { execSync } from 'child_process';
import * as fs from 'fs/promises';
import { AgentTask, CriterionResult, SuccessCriterion } from './types';

export class Scorer {
  async evaluate(task: AgentTask, workDir: string): Promise<CriterionResult[]> {
    const results: CriterionResult[] = [];
    for (const criterion of task.successCriteria) {
      results.push(await this.evaluateCriterion(criterion, workDir));
    }
    return results;
  }

  private async evaluateCriterion(
    criterion: SuccessCriterion,
    workDir: string
  ): Promise<CriterionResult> {
    switch (criterion.type) {
      case 'file_exists':
        return this.checkFileExists(criterion, workDir);
      case 'file_contains':
        return this.checkFileContains(criterion, workDir);
      case 'command_succeeds':
        return this.checkCommandSucceeds(criterion, workDir);
      case 'no_errors':
        return this.checkNoErrors(criterion, workDir);
      case 'llm_judge':
        return this.llmJudge(criterion, workDir);
      default:
        return {
          description: criterion.description,
          passed: false,
          detail: `Unknown criterion type: ${criterion.type}`,
        };
    }
  }

  private async checkFileContains(
    c: SuccessCriterion,
    workDir: string
  ): Promise<CriterionResult> {
    const filePath = c.params.path as string;
    const pattern = c.params.pattern as string;
    try {
      const contents = await fs.readFile(`${workDir}/${filePath}`, 'utf-8');
      const regex = new RegExp(pattern);
      const passed = regex.test(contents);
      return {
        description: c.description,
        passed,
        detail: passed ? 'Pattern matched' : `Pattern "${pattern}" not found in ${filePath}`,
      };
    } catch (err) {
      return {
        description: c.description,
        passed: false,
        detail: `Could not read file: ${(err as Error).message}`,
      };
    }
  }

  private async checkCommandSucceeds(
    c: SuccessCriterion,
    workDir: string
  ): Promise<CriterionResult> {
    const command = c.params.command as string;
    try {
      const output = execSync(command, {
        cwd: workDir,
        stdio: 'pipe',
        timeout: 30000,
      });
      return {
        description: c.description,
        passed: true,
        detail: output.toString().slice(0, 500),
      };
    } catch (err) {
      return {
        description: c.description,
        passed: false,
        detail: `Command failed: ${(err as Error).message}`,
      };
    }
  }

  private async checkFileExists(
    c: SuccessCriterion,
    workDir: string
  ): Promise<CriterionResult> {
    const filePath = c.params.path as string;
    try {
      await fs.access(`${workDir}/${filePath}`);
      return { description: c.description, passed: true, detail: 'File exists' };
    } catch {
      return { description: c.description, passed: false, detail: 'File not found' };
    }
  }

  private async checkNoErrors(
    c: SuccessCriterion,
    workDir: string
  ): Promise<CriterionResult> {
    const command = c.params.command as string;
    try {
      execSync(command, { cwd: workDir, stdio: 'pipe', timeout: 30000 });
      return { description: c.description, passed: true, detail: 'No errors detected' };
    } catch (err) {
      return {
        description: c.description,
        passed: false,
        detail: `Errors found: ${(err as Error).message}`,
      };
    }
  }

  private async llmJudge(
    c: SuccessCriterion,
    workDir: string
  ): Promise<CriterionResult> {
    const rubric = c.params.rubric as string;
    const targetFile = c.params.path as string;
    const contents = await fs.readFile(`${workDir}/${targetFile}`, 'utf-8');

    const judgePrompt = `You are an evaluation judge. Score the following code against this rubric:\n\nRubric: ${rubric}\n\nCode:\n${contents}\n\nRespond with JSON: {"passed": boolean, "reasoning": string}`;

    try {
      const result = execSync(
        `claude -p "${judgePrompt.replace(/"/g, '\\"')}" --output-format json`,
        { stdio: 'pipe', timeout: 60000, maxBuffer: 5 * 1024 * 1024 }
      );
      const parsed = JSON.parse(result.toString());
      const inner = JSON.parse(parsed.result);
      return {
        description: c.description,
        passed: inner.passed,
        detail: inner.reasoning,
      };
    } catch (err) {
      return {
        description: c.description,
        passed: false,
        detail: `Judge failed: ${(err as Error).message}`,
      };
    }
  }
}

The llm_judge criterion type is powerful for assessing code quality, style adherence, or correctness where a regex cannot capture intent. It uses Claude Code itself as the judge, keeping your toolchain consistent.

Orchestrating the Full Evaluation

Now we tie everything together with a main entry point that loads tasks, runs them, and produces a report:

// src/index.ts
import * as fs from 'fs/promises';
import * as path from 'path';
import { HarnessRunner } from './runner';
import { Scorer } from './scorer';
import { AgentTask, TaskResult } from './types';

async function loadTasks(tasksDir: string): Promise<AgentTask[]> {
  const files = await fs.readdir(tasksDir);
  const tasks: AgentTask[] = [];
  for (const file of files) {
    if (file.endsWith('.json')) {
      const contents = await fs.readFile(path.join(tasksDir, file), 'utf-8');
      tasks.push(JSON.parse(contents));
    }
  }
  return tasks;
}

function generateReport(results: TaskResult[]): string {
  const total = results.length;
  const passed = results.filter(r => r.passed).length;
  const passRate = ((passed / total) * 100).toFixed(1);

  let report = `# Agent Evaluation Report\n\n`;
  report += `**Overall Pass Rate: ${passRate}% (${passed}/${total})**\n\n`;
  report += `| Task ID | Status | Duration | Criteria |\n`;
  report += `|---------|--------|----------|----------|\n`;

  for (const r of results) {
    const status = r.passed ? '✅ PASS' : '❌ FAIL';
    const duration = `${(r.durationMs / 1000).toFixed(1)}s`;
    const critSummary = `${r.criteriaResults.filter(c => c.passed).length}/${r.criteriaResults.length}`;
    report += `| ${r.taskId} | ${status} | ${duration} | ${critSummary} |\n`;
  }

  report += `\n## Failure Details\n\n`;
  for (const r of results.filter(r => !r.passed)) {
    report += `### ${r.taskId}\n`;
    for (const c of r.criteriaResults.filter(c => !c.passed)) {
      report += `- **${c.description}**: ${c.detail}\n`;
    }
    report += `\n`;
  }

  return report;
}

async function main() {
  const tasksDir = process.argv[2] ?? './tasks';
  const tasks = await loadTasks(tasksDir);

  console.log(`Loaded ${tasks.length} tasks from ${tasksDir}`);

  const scorer = new Scorer();
  const runner = new HarnessRunner(scoror);

  const results: TaskResult[] = [];
  for (const task of tasks) {
    console.log(`Running task: ${task.id}`);
    const result = await runner.runTask(task);
    results.push(result);
    console.log(`  Result: ${result.passed ? 'PASS' : 'FAIL'} (${result.durationMs}ms)`);
  }

  const report = generateReport(results);
  await fs.writeFile('evaluation-report.md', report);
  console.log('\nReport written to evaluation-report.md');
  console.log(report);
}

main().catch(console.error);

Run the harness with:

npx tsx src/index.ts ./tasks

Best Practices

Keep Tasks Isolated and Deterministic

Every task should start from a known state and not depend on network calls, external APIs, or prior task artifacts. If a task requires a dependency, install it during setup in the temporary directory. This ensures that running the same task twice produces the same baseline conditions.

Start with a Small, High-Signal Suite

It is tempting to write hundreds of tasks, but a suite of 15 to 30 well-chosen tasks is more valuable than 200 shallow ones. Focus on tasks that exercise real agent capabilities: multi-file refactoring, debugging, test generation, and tool orchestration. Each task should probe a distinct capability so failures point to specific weaknesses.

Use Multiple Criterion Types Together

A single task should combine deterministic checks with subjective ones. For example, a refactoring task might check that tests still pass (deterministic), that the target function was modified (file_contains), and that the code remains readable (llm_judge). This layered approach catches both functional and quality regressions.

Version Your Tasks and Agent Configuration

Store task definitions in version control alongside the agent's system prompt, tool definitions, and model version. When you compare evaluation runs across time, you need to know exactly what changed. Tag evaluation reports with the commit hash of both the task suite and the agent configuration.

Set Realistic Timeouts and Turn Limits

Agents can loop indefinitely if they get stuck. Always set both a wall-clock timeout and a --max-turns limit. For most coding tasks, 120 seconds and 15 turns are reasonable starting points. Tasks that consistently hit these limits indicate either an under-specified prompt or a genuine agent limitation worth investigating.

Log Everything for Debugging

Persist the full agent log, environment state, and scorer details for every run. When a task fails, you need to reconstruct what the agent attempted. Consider snapshotting the working directory after execution so you can inspect the final file state manually.

Run Evaluations in CI

Integrate the harness into your CI pipeline so every change to prompts, tools, or model configuration triggers an evaluation run. Fail the build if the pass rate drops below a threshold. This creates a feedback loop that prevents silent regressions from reaching production.

Conclusion

Building an agent evaluation harness with Claude Code gives you a repeatable, quantitative way to measure and improve agent performance. By defining structured tasks, running the agent in isolated environments, and scoring results with both deterministic checks and LLM-based judges, you create a feedback loop that turns vague concerns about agent quality into actionable data. Start small with a focused task suite, integrate it into your development workflow, and expand iteratively as your agent's capabilities grow. The investment pays off the first time a prompt change silently breaks a capability your harness catches before deployment.