regression-evals

Regression Evals

Every prompt change, model upgrade, or tool tweak is a potential regression. Regression evals catch breakage before users do — if you gate deploys on them.

When to Use

• Any AI feature in production
• Before upgrading model versions
• Before merging prompt changes
• Weekly as a drift/decay check

The Core Loop

1. Maintain a versioned eval dataset (see eval-dataset-design)
2. On every proposed change, run evals on baseline AND candidate
3. Compare per-stratum metrics with significance tests
4. Gate merge on: no stratum regresses beyond threshold

Minimal Harness

interface EvalCase { id: string; input: any; expected: any; stratum: string }

async function runEvals(model: string, prompt: string, cases: EvalCase[]) {
  const results = [];
  for (const c of cases) {
    const output = await runModel(model, prompt, c.input);
    const score = await scoreOutput(c.expected, output); // 0-1
    results.push({ id: c.id, stratum: c.stratum, score });
  }
  return results;
}

const baseline = await runEvals("claude-sonnet-4-6", oldPrompt, cases);
const candidate = await runEvals("claude-sonnet-4-6", newPrompt, cases);

const regressed = compareByStratum(baseline, candidate, { alpha: 0.05 });
if (regressed.length) throw new Error(`Regressions in: ${regressed.join(", ")}`);

Significance Testing

A 2% drop on 100 items is noise, not signal. Use bootstrap confidence intervals:

import numpy as np
def bootstrap_ci(scores, n=1000, alpha=0.05):
    means = [np.mean(np.random.choice(scores, size=len(scores), replace=True)) for _ in range(n)]
    return np.percentile(means, [100*alpha/2, 100*(1-alpha/2)])

baseline_ci = bootstrap_ci(baseline_scores)
candidate_ci = bootstrap_ci(candidate_scores)

# If CIs don't overlap AND candidate lower, it's a real regression.

For pass/fail metrics, use McNemar's test on the paired outcomes. For scalars, paired bootstrap.

Thresholds

Hard rules:

Metric	Regression threshold	Action
Any stratum	≥ 3% drop with p < 0.05	Block merge
Aggregate	≥ 5% drop with p < 0.05	Block merge
Single catastrophic item	New case drops from pass→fail	Investigate, likely block
Variance	CI widens significantly	Investigate (noisier outputs)

Soft rules:

• 1-3% drop: warning, review required
• Improvement on one stratum, regression on another: flag for human judgment

CI Integration

# .github/workflows/eval.yml
name: regression-evals
on: pull_request
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: npm ci
      - run: npm run evals:baseline  # fetch cached baseline scores
      - run: npm run evals:candidate
      - run: npm run evals:compare
      - uses: actions/upload-artifact@v4
        with:
          name: eval-report
          path: eval-report.html

Publish the report as a PR comment. Reviewers should see which strata changed and by how much.

Gating Model Upgrades

When Anthropic ships a new model, don't just swap in production. Gate:

1. Run full eval on new model vs current model
2. Bucket regressions: acceptable, acceptable-with-prompt-tweak, hard-regression
3. For hard regressions, either stay on current model or fix before swapping
4. Canary new model to 5-10% of traffic; monitor live metrics for a week before full rollout

Baseline Management

Your "baseline" needs to be a real, stored artifact — not "whatever main is today".

Store baselines in an artifact store keyed on (prompt version, model, dataset version):

evals/baselines/
  prompt-v42_claude-sonnet-4-6_dataset-v3.json

When you merge a change, UPDATE the baseline. New baseline is the reference for the next candidate.

Handling Flaky Evals

Some items are nondeterministic even at temperature 0 (tools, time-dependent data). Options:

1. Mark items as flaky; exclude from regression calc; track separately
2. Run flaky items N times; use majority vote
3. Fix the underlying issue — usually a bad eval item, not a bad model

Don't let flakes erode your alert fidelity.

Alert Fatigue

If every PR triggers an alert, reviewers stop reading. Tune:

• Raise the p-value threshold for borderline cases
• Classify small drifts as "review required" (not blocking)
• Group related strata to reduce notification count

Fewer, higher-quality alerts get acted on.

Handling a Real Regression

When CI blocks the PR:

1. Read the report — which stratum dropped?
2. Look at specific failed items — reproduce locally
3. Is it the prompt, the tool, the model? Bisect if needed
4. Fix and re-run. If you can't fix, decide: accept the regression (with sign-off) or revert

Never:

• Lower the threshold to make it pass
• Remove the failing stratum
• Skip eval on "urgent" PRs — those are exactly when you need it most

Anti-Patterns

1. Running evals once, shipping forever — drift happens
2. No per-stratum breakdown — aggregate hides localized regressions
3. No significance testing — you chase noise
4. Manual compare — human eyeballs miss small systematic drops
5. Baseline updated silently — you lose the ability to detect drift
6. Lowering thresholds to make CI green — defeats the purpose

Best Practices

1. Run regression evals on every PR, gate merges
2. Report per-stratum metrics with confidence intervals
3. Store versioned baselines in an artifact store
4. Use paired bootstrap or McNemar's for significance
5. Canary model upgrades; don't swap in production cold
6. Publish eval reports as PR comments for visibility
7. Maintain flaky-item exclusion lists explicitly, don't hide them