/stdd:mutate — Mutation-test one spec's tests

/stdd:mutate — Mutation-test one spec's tests

Feature: $ARGUMENTS

AC-N tags prove a test exists per criterion; they do not prove it would fail if the code broke. This estimates (or measures) that by mutating the implementation and checking the tests notice. That is the one thing the guard, gate, and audit cannot do. It is opt-in and advisory: it never gates, it is read-only to your working tree, and it is a per-module capstone you run deliberately before merging something critical, not a step in the everyday spec → tdd → review loop.

When to run it

By risk, never on a fixed cadence, and never automated or gated:

• Not per test. The TDD loop already checks each test's teeth: watch every new test fail before you make it pass. A test you saw go RED against the un-implemented code has proven it fails when that behaviour is absent. That is a mutation test. Re-checking it here is redundant, and mutation testing needs a GREEN tree anyway (there's nothing coherent to mutate mid-loop).
• Not on every slice. A full run re-executes the suite once per mutant, minutes even in sampling mode. /stdd:review already tells you a tagged test exists per criterion, cheaply. mutate answers the more expensive question: do those tests have teeth? And only that.
• Yes before merging a critical module. A security boundary, a money path, a parser, a durability guarantee: code where a silently-weak assertion is expensive. That's the trigger: stakes, not rhythm. Specs carry this judgment in their **Criticality:** critical field (set at /stdd:spec), and /stdd:tdd offers this command at Finish precisely for those slices, so the timing is delegated to stdd while the run stays opt-in.

Scope: exactly one spec (this is a hard rule)

Mutation testing is per-module by design, and a full run is expensive: one test run per mutant. Resolve one target and stop there:

• $ARGUMENTS must name exactly one spec. Strip the --exhaustive flag first; what remains is the feature name.
• If it is empty, names more than one spec, or is all / * / similar: stop and ask the user which single spec to test. Never fan out across every spec. That is what turns a 2-minute check into a multi-hour run that exhausts disk and CPU. List the candidate specs and let them pick one.
• Read docs/specs/<name>.md, locate its implementation file(s) under src/ and its AC-N-tagged test(s). If no spec or no tests exist, stop: there is nothing to mutate against. Scope everything below to that one module.

Mode

• Sampling (default): fast, polyglot, zero new dependencies. A sub-agent proposes a handful of spec-tied mutations and explains the gaps. Use for a quick read on whether a spec's tests have teeth. This is the blessed default.
• Exhaustive (--exhaustive): a power path. Runs a real mutation framework over the module for a complete, repeatable mutation score. Slower, needs a per-language tool. Use before merging a critical module, or to track a score over time.

Both modes run isolated and touch nothing in your tree. See below.

Sampling mode (default)

Spawn a sub-agent (Task tool) with isolation: worktree so it edits an isolated throwaway copy and never your tree. (The Write guard still applies in that worktree, but it only gates new src/ files; mutating existing source is allowed.) Brief:

For the module(s) implementing docs/specs/$ARGUMENTS.md, generate ≈5–8 behaviourally meaningful mutations, each tied to an acceptance criterion: flip a boundary (<→<=), drop a negation, weaken a guard, return empty/null, swap a branch, off-by-one. Skip mutations you judge equivalent (unobservable through the public interface) and say why. For each: apply it, run only this spec's tagged tests via ./scripts/run-tests.sh --all <test-file>, record killed (a test failed) or survived (all passed). Revert (git checkout the file) between mutations, and end with a clean worktree. Return JSON: { mutants: [{ ac, location, change, result, note }], summary }.

If one spec spans several source files, you may run one sub-agent per file of that spec in parallel, but stay within this single spec. Do not spawn agents for other specs.

Exhaustive mode (--exhaustive)

Exhaustive mode must be just as isolated as sampling: it installs a framework and writes config, which must never land in the user's tree. Do it one of two ways, in order of preference:

1. Worktree sub-agent (preferred). Spawn a sub-agent with isolation: worktree, brief it to install the framework, write the config, run it, and return the report JSON. The worktree (under .claude/worktrees/, disk-backed) is auto-cleaned; your tree (including package.json / lockfiles) is never modified.
2. In-place only on a confirmed-clean tree. If you cannot use a worktree, first verify git status --porcelain is empty; if the tree is dirty, stop and tell the user to commit or stash. Run, then fully revert: remove the config, uninstall the framework, and delete its artifacts (reports/, .stryker-tmp/, logs). Confirm git status is clean at the end.

Never leave the framework, its config, or its artifacts in the user's tree.

Pick the tool by framework and scope mutate to the spec's module only:

• Bun / Node (TS/JS) → Stryker. Test runner command, pointed at the unit tier for speed:

  // stryker.config.json (scope `mutate` to THIS spec's files only)
  { "testRunner": "command",
    "commandRunner": { "command": "./scripts/run-tests.sh" },
    "mutate": ["src/<module>/**/*.ts", "!src/**/*.test.ts"],
    "concurrency": 2,
    "tempDirName": ".stryker-tmp" }
  

  Run with the Node runtime, not bunx --bun, which crashes in the Stryker instrumenter (@babel/generator CJS interop: TypeError: generator is not a function):

  bunx stryker run          # resolves to Node; do NOT pass --bun
  

  (deps: @stryker-mutator/core.)

• Python (pytest) → mutmut. mutmut run --paths-to-mutate src/<module> (it reverts each mutation; clean tree matters). mutmut results lists survivors. (cosmic-ray is the heavier alternative.)

If the tool is not installed/configured, do not silently skip: print the exact devDependency + config + command above and ask the user whether to add it: exhaustive testing is opt-in and pulls a real dependency.

Environment guardrails

A full mutation run re-runs the whole test suite once per mutant, so it is I/O- and CPU-heavy and its scratch usage is amplified hundreds of times:

• Run one repo at a time. Do not launch /stdd:mutate --exhaustive in several repos simultaneously. Concurrent runs contend for CPU and share the same scratch filesystem.
• Keep scratch off a RAM tmpfs. If /tmp is a small RAM-backed tmpfs (common on WSL, often 16 GB with a 1M-inode cap), point the run's TMPDIR at a disk-backed dir so the suite's own mkdtemp scratch and the framework sandbox don't exhaust it mid-run (which deadlocks tooling, even rm fails once the device is full). E.g. TMPDIR="$PWD/.stryker-tmp/os" bunx stryker run.
• Cap concurrency ("concurrency": 2 above) rather than the default, and scope mutate to the one module: do not mutate all of src/.

Report (both modes)

• A table: AC · mutation/location · result (killed / survived / equivalent).
• Split survivors honestly by test tier. The command runner runs ./scripts/run-tests.sh = the unit tier only. Code exercised only by the integration/e2e tier will show as survivors even though real integration tests would kill them: those are the measurement's blind spot, not unit-suite gaps. Separate them into two groups: (a) integration-only blind spots (expected survivors, name the tier that covers them) and (b) genuine unit-suite gaps.
• For each genuine survivor, name the missing/weak assertion and the concrete test to add, the point of the exercise.
• A summary line: sampling → N/M sampled mutants killed (estimate); exhaustive → the framework's mutation score: X% (trendable). State plainly that the score is unit-tier-only.

Stop at the report — do not fix

This command measures and stops. Do not modify the spec, tests, or source in your tree, and do not commit, not even to "close a survivor." To act on a gap, hand the user to /stdd:tdd $ARGUMENTS: add the missing assertion test-first there, in a separate, gated turn. mutate's only artifact is the survivor list; turning it into edits is a different command's job.

Limits

Sampling is a spot-check, not proof: the model chooses the mutations, so coverage isn't systematic. A clean run is evidence, not a guarantee, and its "equivalent" calls are judgement, not fact. Exhaustive is complete for the mutation operators the tool implements and gives a real, trendable score. But it costs one test run per mutant (scope to the module, run the unit tier), adds a per-language dependency, and its score is only as honest as the tier you run. Neither gates commits.