verification-loop

Verification Loop

Overview

Verify after each small change—not only at the end. Two modes share the same discipline:

1. Routine verification — run the right checks for the change type.
2. Claim-based verification — prove or disprove a specific, measurable claim with baseline vs treatment evidence.

When to Use

• After every code, dependency, or configuration change
• Before every commit or merge
• When the user asks to "verify this", "prove it works", or "show evidence"
• When tests pass but user-visible behavior still needs confirmation

Do not use claim-based mode for vague assertions like "the code is cleaner". Ask for a falsifiable claim first.

The Loop

Change → Verify → If fail: Fix → Verify → If pass: Continue

Step 1: Change

Make a small, atomic change (~100 lines max).

Step 2: Verify

Pick the mode that fits:

Situation	Mode	Action
Normal development	Routine	Run checks from the table below
Specific claim to prove/disprove	Claim-based	Follow the claim workflow below

Routine checks by change type:

Change Type	Verification
Code change	Tests for affected area
New feature	New tests + existing tests
Bug fix	Regression test + existing tests
Refactor	All tests (behavior must not change)
Dependency	Build + tests + security scan
Configuration	Build + smoke test

Claim-based workflow:

1. Restate the claim: condition, metric, threshold.
2. Pick the smallest surface that can disprove it.
3. Capture baseline (merge base, parent commit, or broken repro).
4. Capture treatment with the same command, data, warmup, and environment.
5. Compare artifacts: numbers, screenshots, transcripts, HTTP responses, profiles, heap snapshots, test output.
6. Return exactly one verdict: VERIFIED, NOT VERIFIED, or INCONCLUSIVE.

Surfaces for claims: unit/integration tests, control-cli / control-ui, browser traces, local HTTP/RPC diffs, timings, heap snapshots.

Verdict rules:

• VERIFIED — predicted change met threshold, no obvious confound
• NOT VERIFIED — unchanged, wrong direction, or below threshold
• INCONCLUSIVE — invalid baseline, noisy signal, or environment mismatch

Claim output shape:

VERIFIED | NOT VERIFIED | INCONCLUSIVE
Claim: <falsifiable claim>
Evidence: <metric>: baseline=..., treatment=..., delta=..., threshold=...
Reasoning: <one paragraph>

Step 3: Fix (if failed)

Read error output, fix the specific issue, do not change unrelated code, return to Step 2.

Step 4: Continue (if passed)

Commit, move to the next change, repeat.

Verification Levels

Level	Scope	When
Unit	Affected function/module	After every small change
Integration	Affected feature	After feature complete
Full	All tests	Before commit/PR
E2E	Critical user flows	Before release

Commands

# TypeScript/JavaScript
bun test && bun run build && bun run lint && bun run typecheck

# Python
pytest && ruff check . && mypy .

# Go
go test ./... && go vet ./... && golangci-lint run

# Rust
cargo test && cargo clippy && cargo build

Anti-Rationalization

Excuse	Counter
"I'll test everything at the end"	Errors compound. Verify early, verify often.
"The tests take too long"	Run affected tests first; full suite before commit.
"This change is too small to test"	Small changes can have large effects.
"It works on my machine"	Local verification is the minimum; CI validates the target environment.
"It probably works"	Without a falsifiable claim and comparison, you do not know.