eval-golden-set-builder

Eval Golden Set Builder

When to invoke

• Starting evals from scratch on a new prompt or agent.
• An existing golden set is producing noisy deltas (every change looks "better").
• A product change (new tool, new model, new system prompt) needs a before/after regression check.
• LLM-judge results are inconsistent across runs.

Step 1 — Define what you are measuring

Before collecting examples, answer these:

Question	Why it matters
What is the unit of success?	A correct final answer? A correctly-chosen tool? A well-structured JSON output?
Who is the judge? (human / programmatic / LLM)	Determines reference-answer format
What is the blast radius of a regression?	Determines minimum set size and grader strictness

One eval, one signal. Do not try to evaluate "everything" in one set. Separate factual accuracy from tone from tool-call correctness — they need different graders.

Step 2 — Sampling strategy

Stratum	Target fraction	What goes here
Happy path	40 %	Canonical inputs the system handles today
Edge / boundary	30 %	Near-miss inputs, ambiguous phrasing, missing fields
Adversarial	20 %	Injection probes, jailbreak-adjacent inputs, known failure modes
Regression seeds	10 %	Any input that caused a past production bug

Minimum viable set: 50 examples for a binary pass/fail grader; 100+ for a scored (0–5) rubric or LLM judge (smaller sets produce deltas too noisy to trust).

Step 3 — Authoring reference answers

1. Human-authored ground truth first. Write the ideal answer before you see what the model produces — post-hoc rationalization corrupts the set.
2. For tool-call evals: record the expected tool_use block — name + exact input keys. Use "contains" matching (not string equality) for free-text arguments.
3. For structured-output evals: store the expected JSON; use JSON Schema validation as the grader, not string diff.
4. For open-ended answers: write a rubric (3–5 criteria, each scored 1–3) that a judge (human or LLM) can apply without seeing the question first. Store it alongside the example.

{"id": "q_001", "input": {"messages": [...]}, "expected": {"tool": "search_docs", "query_contains": "refund policy"}, "grader": "tool_call_match"}
{"id": "q_002", "input": {"messages": [...]}, "expected": {"rubric": "criteria_accuracy_helpfulness"}, "grader": "llm_rubric"}

Step 4 — Grader pairing

Output type	Grader	Notes
Exact JSON / schema	JSON Schema validator	Fast, deterministic, free
Tool call correctness	Name + key-subset match	Allow partial input matches
Binary pass/fail factual	String in / regex	Hand-write assertions
Scored rubric / tone / style	LLM-as-judge (Haiku via Batch)	Run via Anthropic Batch at 50 % cost; randomize answer order per call to reduce position bias

LLM-judge discipline: present judge with rubric + response only (not the question) to prevent question-loading bias. Run 3× and take majority vote on borderline cases.

Step 5 — Baseline and delta protocol

# Run on every prompt/model/tool change
baseline = run_eval(golden_set, old_config)
candidate = run_eval(golden_set, new_config)
delta = candidate.mean_score - baseline.mean_score
p_value = ttest(candidate.scores, baseline.scores).pvalue
print(f"delta={delta:+.3f}  p={p_value:.3f}  n={len(golden_set)}")
# Merge only if delta >= +0.02 and p < 0.05

Treat a delta < 0.02 on a 0–1 scale as noise, not improvement. Track per-stratum scores — an adversarial regression hidden by a happy-path improvement is still a regression.

Pitfalls

• Seeding the golden set from model output ("label what the model said as correct") — creates a circular reference that only confirms the current model's biases.
• A set with 90 % happy-path inputs — edge and adversarial cases are where regressions actually hide.
• Comparing runs on different dates without pinning the model version and temperature (temperature: 0 for evals).
• Re-running a failing test until it passes and calling that "fixed" — add the fixed case as a regression seed; don't delete the test.