Golden Datasets: How to Build and Maintain Eval Sets

If you ship machine learning or LLM-based features, you've probably felt the pain of "vibes-based" evaluation. You tweak a prompt, run it on three examples that come to mind, and convince yourself it's better. Then a user reports a regression in production. The fix isn't more intuition — it's a golden dataset: a curated, versioned set of input-output pairs that serves as the ground truth for measuring whether your system is actually improving.

This tutorial walks through what golden datasets are, why they matter, how to build one from scratch, and how to keep it healthy as your product evolves. We'll include runnable code examples using Python and common tooling.

>>> NEW: The Self-Hosted Agent Factory — Production Kit

Run a 3-agent production line (orchestrator + workers + adversarial QC) on a $6/mo VPS. 120-page book + the complete working Docker stack: task ledger, model router, 15-min watchdog, QC rubric.

Launch price $29 (first 100 copies, then $59) →

What Is a Golden Dataset?

A golden dataset (also called an eval set, reference set, or benchmark set) is a collection of examples where each entry contains:

An input — the prompt, query, or feature vector your system receives.
An expected output — the canonical answer, label, or behavior you want.
Metadata — tags, categories, difficulty levels, source, and notes that help you slice results.
Optionally, rubrics — criteria an LLM judge or human reviewer should use to score the output.

Unlike training data, a golden dataset is not consumed by the model. It is held out and used purely to measure performance. Think of it as the unit test suite for your AI system.

Why Golden Datasets Matter

Without a golden dataset, every change is a gamble. With one, you get:

Reproducible measurement — the same inputs, scored the same way, every release.
Regression detection — a fix that helps one case but breaks three others becomes visible immediately.
Confidence to ship — a passing eval run is evidence, not opinion.
Shared language — product, engineering, and QA can debate specific examples instead of vague feelings.
Targeted improvement — slicing by category tells you exactly where to invest.

The cost of building one is real, but it pays back every single time you change a prompt, swap a model, or adjust retrieval parameters.

How to Build a Golden Dataset

Step 1: Define What You're Evaluating

Before collecting examples, write down the behaviors you care about. For a RAG chatbot, that might be: factual accuracy, citation correctness, refusal appropriateness, and conciseness. Each behavior becomes a dimension you can score.

Step 2: Source Diverse Examples

The biggest mistake teams make is building a tiny, easy eval set. Aim for a mix of:

Common cases — the 80% of traffic you must get right.
Edge cases — ambiguous queries, empty inputs, adversarial prompts.
Production samples — real user queries, anonymized and labeled.
Known failure cases — every bug report should spawn a golden example.
Adversarial cases — prompts designed to break safety or correctness.

A reasonable starting size is 100–300 examples for an LLM feature, growing over time. Quality and coverage matter far more than raw count.

Step 3: Create the Reference Outputs and Rubrics

For each input, produce the ideal output. For tasks with a single correct answer (classification, extraction), this is straightforward. For generative tasks, you may need rubrics instead of exact matches:

{
  "id": "eval_0042",
  "input": "Summarize the refund policy in two sentences.",
  "context_docs": ["doc_refund_policy_v3"],
  "expected": "Customers can request a full refund within 30 days of purchase. Refunds are processed to the original payment method within 5 business days.",
  "rubric": {
    "factual_accuracy": "Every claim must be supported by the context docs.",
    "length": "Exactly two sentences.",
    "no_hallucination": "No details outside the provided context."
  },
  "category": "summarization",
  "difficulty": "easy",
  "source": "production_sample",
  "created_at": "2025-01-15"
}

Step 4: Store It as Versioned Data

Treat your golden dataset like code. Store it in your repository as JSONL, commit changes through pull requests, and tag releases. Here's a simple loader:

import json
from dataclasses import dataclass
from pathlib import Path
from typing import Any

@dataclass
class GoldenExample:
    id: str
    input: str
    expected: str
    rubric: dict[str, Any]
    category: str
    difficulty: str
    source: str

def load_golden_dataset(path: str | Path) -> list[GoldenExample]:
    examples = []
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            row = json.loads(line)
            examples.append(GoldenExample(
                id=row["id"],
                input=row["input"],
                expected=row.get("expected", ""),
                rubric=row.get("rubric", {}),
                category=row.get("category", "uncategorized"),
                difficulty=row.get("difficulty", "unknown"),
                source=row.get("source", "manual"),
            ))
    return examples

dataset = load_golden_dataset("golden_dataset.jsonl")
print(f"Loaded {len(dataset)} golden examples")

Step 5: Run Your System Against the Eval Set

Here's a minimal evaluation harness that calls your system and scores each example:

from dataclasses import dataclass, field

@dataclass
class EvalResult:
    example_id: str
    passed: bool
    score: float
    output: str
    notes: str = ""

def run_system(prompt: str) -> str:
    # Replace with your actual LLM / model call
    return "Customers can request refunds within 30 days."

def exact_match_score(expected: str, output: str) -> float:
    return 1.0 if expected.strip().lower() == output.strip().lower() else 0.0

def evaluate(dataset, scorer=exact_match_score) -> list[EvalResult]:
    results = []
    for ex in dataset:
        output = run_system(ex.input)
        score = scorer(ex.expected, output)
        results.append(EvalResult(
            example_id=ex.id,
            passed=score >= 0.99,
            score=score,
            output=output,
        ))
    return results

def summarize(results: list[EvalResult]) -> dict:
    total = len(results)
    passed = sum(1 for r in results if r.passed)
    avg_score = sum(r.score for r in results) / total if total else 0.0
    return {
        "total": total,
        "passed": passed,
        "pass_rate": round(passed / total, 4) if total else 0.0,
        "avg_score": round(avg_score, 4),
    }

results = evaluate(dataset)
print(summarize(results))

Step 6: Use LLM-as-a-Judge for Generative Tasks

Exact match rarely works for open-ended outputs. A common pattern is to use a strong model as a judge, guided by your rubric:

import openai

def llm_judge_score(example: GoldenExample, output: str) -> float:
    rubric_text = "\n".join(f"- {k}: {v}" for k, v in example.rubric.items())
    judge_prompt = f"""You are an evaluator. Score the OUTPUT against the EXPECTED answer
using the following rubric. Return only a JSON object with "score" (0.0 to 1.0) and "reason".

RUBRIC:
{rubric_text}

INPUT:
{example.input}

EXPECTED:
{example.expected}

OUTPUT:
{output}
"""
    response = openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": judge_prompt}],
        response_format={"type": "json_object"},
        temperature=0,
    )
    import json
    parsed = json.loads(response.choices[0].message.content)
    return float(parsed["score"])

results = evaluate(dataset, scorer=lambda exp, out: llm_judge_score(exp, out))
print(summarize(results))

Calibrate your judge against human labels on a small subset first. LLM judges drift, so re-validate periodically.

Step 7: Slice and Analyze

Aggregate scores are useful but hide problems. Slice by category and difficulty:

from collections import defaultdict

def slice_results(results, dataset):
    by_id = {ex.id: ex for ex in dataset}
    slices = defaultdict(list)
    for r in results:
        ex = by_id[r.example_id]
        slices[ex.category].append(r)
        slices[f"difficulty:{ex.difficulty}"].append(r)

    report = {}
    for name, group in slices.items():
        report[name] = summarize(group)
    return report

for slice_name, stats in slice_results(results, dataset).items():
    print(f"{slice_name:30s} {stats}")

This is where you discover that your summarization pass rate is 95% overall but 40% on adversarial inputs — the insight that drives real improvement.

Best Practices for Maintaining Golden Datasets

Grow It Continuously

Every production bug, user complaint, and edge case you discover should become a new golden example. Set a target like "add 5 examples per week" and treat it as a recurring engineering task, not a one-off project.

Version Everything

Tag your dataset alongside model and prompt versions. When you compare two runs, you must know which dataset revision each used. A simple convention: dataset_v{N} tags in git, recorded in your eval logs.

Avoid Overfitting to the Eval Set

If the same people writing prompts also hand-craft every golden example, you risk optimizing for the eval set rather than real users. Mitigate this by:

Drawing examples from real production traffic.
Rotating in fresh examples and retiring stale ones.
Keeping a held-out "shadow" set you only score periodically.

Keep It Clean

Golden examples go stale. Products change, policies update, expected answers shift. Schedule a quarterly review where you:

Re-label examples whose expected outputs are now wrong.
Remove duplicates and trivially easy examples that always pass.
Promote hard production cases into the set.

Automate the Eval Run

Wire evaluation into CI so every prompt or model change triggers a run. Block merges on regressions above a threshold:

# .github/workflows/eval.yml (conceptual)
name: Eval
on: [pull_request]
jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: python eval_pipeline.py --dataset golden_dataset.jsonl --threshold 0.85
      - run: python eval_report.py --compare main --fail-on-regression

Document Labeling Conventions

Write a short README describing how expected outputs are produced, how rubrics are written, and how disagreements are resolved. New contributors should be able to add examples consistently without guessing.

Conclusion

A golden dataset is the single highest-leverage investment you can make in an AI-powered product. It transforms evaluation from subjective debate into measurable progress, catches regressions before they reach users, and gives every prompt tweak a clear pass/fail signal. Start small — even 50 well-chosen examples beat 5,000 noisy ones — and grow the set deliberately alongside your product. Treat it as living infrastructure: version it, review it, automate it, and let it compound. The teams that ship reliable AI features are not the ones with the most clever prompts; they are the ones who know, with evidence, when their system is actually getting better.