running-benchmarks

Running Benchmarks

Overview

You are the machine the producer ran so a perf regression wouldn't reach a human — or production — first. Benchmarks are backpressure: they say "no" to a slow patch at the boundary where it was introduced, not three releases later when someone files a latency bug.

Core principle: a benchmark number is meaningless on its own. A single "after" run proves nothing. A verdict requires a baseline to compare against and a way to separate a real change from run-to-run noise. Everything below exists to enforce those two things, because the natural temptation is to run once and eyeball it — and that is worthless.

When to Use

• A backpressured loop is on a performance-sensitive project and needs to confirm a patch didn't regress perf — a quick targeted check each iteration, and the full suite before leaving the loop.

Not for: projects with no benchmark suite or no perf sensitivity — skip benchmarking entirely rather than inventing it. This is not a correctness check; tests cover that.

Step 1 — Discover the suites and their time budgets

If the project has a BACKPRESSURE.md, check it first — it may name the suite(s) and exact commands to use. Otherwise find the commands yourself — package.json scripts, Makefile, a benchmarks//bench/ dir, CI config, README. Identify which suites exist and roughly how long each takes. Expect a spread: fast micro/targeted suites (seconds) and full end-to-end suites (minutes). Note what each one actually measures so you can map a suite to the code path you touched.

Step 2 — Pick the suite for the moment

• Each iteration: run the fast, targeted suite that covers the path you changed. It must be cheap enough to run after every patch without rabbit-holing.
• Before done: run the full suite once. Targeted-green is necessary, not sufficient — a change can help one path and regress another.

Don't run an 8-minute end-to-end suite to check a change that only touches one micro path; don't declare done off a micro suite alone.

Step 3 — Always measure baseline vs patched

Never judge from the patched numbers alone. Run the unchanged baseline (git stash, or check out the parent commit / a worktree) and your change back-to-back, on the same machine under the same conditions. The baseline is the only thing that makes the "after" number mean anything.

Quiet the machine first — close background load, don't run benchmarks mid-build or while other heavy processes compete. Uncontrolled machine state is the largest real source of noise, and "same conditions" means actually the same, not just same hardware.

Step 4 — Beat the noise with repetition

Benchmark output is noisy (often ±3–5% run to run). One run per side is not evidence.

• Run each side multiple times (≥5 when the suite is cheap).
• Compare the median (and the min for CPU-bound microbenchmarks — least contended, most stable), not a single sample.
• Establish the noise band from the spread of the repeated runs (or a known per-suite figure). The band is your threshold for everything below.

Step 5 — The verdict: regression / improvement / wash

Verdict	Condition	Action
Wash (neutral)	Patched median within the noise band of baseline; distributions overlap	Passing. Move on.
Improvement	Patched consistently faster, gap clears the noise band	Passing. Note the win.
Regression	Patched consistently slower, gap clears the noise band, distributions don't overlap	FAIL. Fix it, or stop and name the blocker — never wave it through.
Ambiguous	Result straddles the band (e.g. ~noise ± a little)	Not a verdict yet. Add runs (≥10/side) and/or run the full suite to resolve before deciding.

If a suspicious result still won't separate from noise after more runs — a small true shift that the suite genuinely can't resolve — do not default it to a pass. Treat it as a blocker and name it (the change may need a less noisy benchmark, or a human call), exactly as you would any other unresolved check.

Step 6 — Record the evidence

"No regression" asserted with no numbers is not a passing check. Record baseline vs patched, run count, and the median/min you compared. Evidence before assertion — the same standard the rest of the loop holds you to.

Common rationalizations

Rationalization	Reality
"I ran it once, the number looks fine"	One run inside a ±5% noise band tells you nothing. Baseline + repeats or it doesn't count.
"It's a tiny bit slower, probably noise"	"Probably noise" is a measurement you didn't take. If it straddles the band, add runs until it resolves.
"The micro suite is green, we're done"	Micro-green can hide a full-suite regression. Run the full suite before done.
"Running the baseline too is double the work"	Without a baseline the patched number is uninterpretable. Half the work producing nothing is not a saving.
"I'll just trust the full suite once at the end"	A regression found at the end is a bisect. Run the targeted suite each iteration.

Red flags — STOP

• Reporting a benchmark result with no baseline to compare against.
• Declaring "no regression" with no recorded numbers.
• Calling an ambiguous straddle-the-band result a pass instead of adding runs.
• Skipping the full suite before done because a targeted suite was green.
• Treating a real, beyond-noise slowdown as acceptable without fixing it or naming the blocker.