captain-obvious

Captain Obvious 🫡

AI agents write tests that can never fail. This skill deletes them.

Every AI-generated test suite has them:

assert isinstance(add(1, 2), int)   # the function is typed `-> int`. mypy knew.

expect(true).toBe(true);            // bold claim

Tests like these can't catch a regression — they re-assert what the compiler already proved, or assert nothing at all. They burn CI minutes and inflate coverage into false confidence. Empirical studies find test smells in 38–100% of LLM-generated test suites.

🧠 The idea: ask the type checker, not another LLM

Detection is fully deterministic — no LLM in the loop, one script invocation per repo. Instead of asking an AI "is this test useful?", it asks the tools that already know:

• TypeScript: the project's own compiler API answers "what is the static type of this expression?" — if typeof result === 'number' is a fact of the type, the assertion is not a test. Plain-JS test files (.test.js etc.) get the syntactic categories; type-guaranteed needs TypeScript (or stays off without checkJs).
• Python: batched mypy reveal_type probes (one mypy run for the whole repo) answer the same for isinstance(...) and is not None.
• AST passes catch the rest: tautologies, mock-echo tests, assertions after a return, assertions swallowed by empty catch/except: pass, duplicate bodies, len(x) >= 0.

✅ Proven vs ⚠️ advisory

Every finding is one of two tiers, and each tier is handled by whoever can actually decide it:

• proven — cannot fail, by construction. Auto-deleted by --fix, no LLM involved. The scripts guard every known way types lie: any/unknown, as casts, non-null !, index signatures, unchecked index access, structural instanceof (only nominal classes are provable), cast()/type: ignore.
• advisory — almost certainly useless but not provable (structural instanceof, mock-echo variants, pytest.raises(Exception), unawaited async assertions, rotten-green conditional asserts). The script writes down exactly why it's uncertain, and the coding agent running the skill adjudicates each one against the surrounding code — delete, keep, or rewrite — then proposes the calls for your approval before touching anything.

It also knows what not to flag: toBeDefined() on .find() results (T | undefined — a real check), enum contract locks, custom assertion helpers, deliberate determinism tests, and "must not raise" contract tests.

📊 Real-world results

• Internal repo, ~1,800 pytest tests: 124 lines deleted across 21 files — 111 assertions re-checking what mypy already guaranteed, plus one all-constant test. Type-checker and full CI green before and after; change auto-approved.
• TypeScript CLI repo, ~730 tests: zero deletions (it doesn't invent work) — but it caught a real bug: a .rejects.toThrow(...) never awaited, so it silently never ran.

The honest headline isn't a line count — it's that --fix only removes what it can prove, so whatever it deletes could never have caught a regression. Clone it and run it on your own repo.

🚀 Install & use

npx skills add shmulc8/captain-obvious@captain-obvious

Also ships a .claude-plugin/plugin.json manifest, or copy skills/captain-obvious/ into ~/.claude/skills/. Then ask your agent to "clean up the useless tests" — or run the scanners directly:

# report-only (add --json out.json to save)
node    skills/captain-obvious/scripts/captain_obvious_ts.mjs --project <repo>
python3 skills/captain-obvious/scripts/captain_obvious_py.py  --path <repo> --mypy "uv run mypy"

# delete proven findings (needs a clean git tree — review the diff after)
node ... --fix

# confirm rotten-green asserts against real coverage (lcov / istanbul / coverage.py)
node    ... --coverage coverage/lcov.info
python3 ... --coverage coverage.json

With --coverage, a conditional-assert whose line never executed is promoted to proven rotten; one that did execute is a confirmed false positive and is dropped — the dynamic half of the ICSE'19 analysis, using coverage your runner already emits.

Trust boundary: the scan runs the project's own toolchain — mypy (with whatever plugins the repo's config loads), uv/poetry environments, the repo's own typescript package. Treat scanning like running the repo's type checker: only do it on code you trust.

Requires Python ≥ 3.9 for the Python scanner and hook; the TS scanner uses the target repo's own typescript and requires ≥ 4.0.

🛡️ Prevention (write-time hook)