stage6-variance-aware-decision

Stage 6 · Iterate · Variance-aware decision (at scale)

The full-scale counterpart of stage3-multi-seed-variance. After multi-seed runs at the target size, decide whether the variant is genuinely better than the baseline.

Stage question

"Has this change produced a real, statistically meaningful improvement at the target scale?"

This is what the project is for; don't get it wrong by under-seeding or over-claiming.

The protocol (same as Stage 3, larger budget)

1. Run baseline arm B with N_B seeds at the target scale (typically N_B ≥ 3).
2. Run variant arm V with N_V seeds at the target scale.
3. Pick the headline metric, ideally agreed on before running.
4. Compute mean and std for each arm.
5. Apply the decision rule.

The decision rule is identical to Stage 3:

• |mean_V − mean_B| > 2 × pooled_std → improvement is probably real.
• Otherwise: indistinguishable.

The difference vs. Stage 3 is:

• Larger budget → can usually afford more seeds.
• Stakes are higher → tendency to confirmation-bias the conclusion. Resist it.
• Final reportable claim → wording and confidence intervals matter.

How many seeds at scale?

Target seeds depends on the variance and the effect size:

• N = 3 → minimum for any directional claim. Confidence is loose.
• N = 5 → standard for a defensible internal claim.
• N = 10+ → standard for a publishable claim.

Don't accept N = 1 per arm at scale. If budget forces N = 1, frame the result as "preliminary" and explicitly state the variance is unmeasured.

Reporting language

Wording is part of the rigor. Use:

• Strong claim (justified by data): "V outperforms B (Δ = +0.018 ± 0.005, 5 seeds each, |Δ| > 2σ)."
• Weak claim (where data warrants weakness): "V appears to outperform B (Δ = +0.012 ± 0.011, 3 seeds each, gap within trial variance)."
• Refutation (when V didn't win): "V did not significantly outperform B (Δ = −0.003, 5 seeds each)."
• Insufficient evidence: "Variance not measured; single seed comparison is uninformative."

Avoid: "V is better" (no quantification), "V tends to be better" (weasel without numbers).

Edge cases

Different seed counts per arm

Use Welch's t-test for unequal sample sizes. The 2-sigma rule still works as a coarse approximation when one arm is much smaller (e.g. N_B = 5, N_V = 3).

One seed diverged

A run that hit kill-criterion is not a "result" — it's a divergence. Two reasonable choices:

1. Exclude the diverged seed and report the remaining ones (state explicitly: "1/5 seeds diverged, excluded").
2. Report the divergence rate as part of the verdict ("V diverged in 2/5 seeds, B diverged in 0/5; this is a stability regression even though successful runs are equivalent").

The right choice depends on whether divergences are due to bad luck or are an intrinsic property of the variant.

Non-Gaussian metric distribution

Some metrics (rare-event accuracy, BLEU on small test sets) are not approximately Gaussian. Use a non-parametric test (Mann-Whitney U) instead of t-test. scipy.stats.mannwhitneyu is the standard choice.

Procedure when assisting a user

1. Reject one-seed-per-arm comparisons. Tell the user the comparison is variance-blind and refuse to issue a verdict.

2. If multi-seed runs exist, compute mean, std, Δ, pooled std, and decision threshold.

3. Render a one-line verdict using the wording table above. Pick the wording that the data justifies, not the wording the user wants.

4. Surface secondary regressions: throughput delta, memory delta, divergence rate. A "marginally better loss but 30% slower" is often net-worse.

5. Recommend follow-up: if marginal, do another N seeds; if conclusive, archive the result and move on.

Boundaries

• This is not a generalization claim — only that the gap is statistically significant on the held-out set used. Generalization to other tasks/domains requires separate evaluation.
• This is not a causal claim about why the improvement happens. For mechanism, see stage3-surrogate-task (if not already done).
• The 2σ rule is a heuristic. For high-stakes decisions, run Welch's t-test or Mann-Whitney U formally.

Common mistakes

• "Best-of-N" reporting → selection bias inflates apparent effect.
• Single-seed comparisons → no variance estimate, no claim.
• Cherry-picking the metric after seeing data → invalidates the test.
• Conflating "didn't reach significance" with "no effect" → it might reach significance with more seeds.

Related

• skills/stage3-multi-seed-variance — same logic at small scale.
• skills/runs-diff — produces the verdict from journal data.
• skills/runs-diff — slash command that wraps this analysis.
• skills/stage6-ablation-matrix — applies the decision across a set of variants, not just two arms.
• Welch's t-test, Mann-Whitney U.