systematic-debugging

Execute

→ Bug? Test failure? Unexpected behavior? → Find root cause first. No fixes without evidence.

1. Isolate: read error → reproduce → check git diff → drill down through diagnostic layers: L1 symptom → L2 logic → L3 system → L4 architecture → L5 cross-system contract → L6 platform constraint → L7 spec gap. Stop when no deeper "why" remains OR terminal unactionable (T1-T4).
2. Identify owner: compare with working code → locate canonical owner → flag duplicate owners as a finding
3. Prove: one hypothesis → minimal test → iterate. 3+ failed fixes = question architecture, do not attempt #4. After fix, if any symptom persists → differential diagnosis (Phase 4 Step 4bis).
4. Fix: failing test → minimal code at canonical owner → verify → Reflection + architecture review → repair + retirement track → Done when: confidence ≥ B, both tracks explicit, DeeperCause answered "no" with evidence, no H-class hard signal still active.

Systematic Debugging

Overview

Random fixes waste time and create new bugs. Symptom fixes are failure.

This skill is the canonical debugging workflow. Use it to move from symptom to root cause, then to the smallest justified fix and retirement plan.

When to Use

Any technical issue: test failures, bugs, unexpected behavior, performance problems, build/integration failures.

Especially under time pressure, when "just one quick fix" seems obvious, after multiple failed fixes, or when duplicate owners / fallback chains may be involved.

The Four Phases

Phase 1: Root Cause Investigation

BEFORE attempting ANY fix:

1. Read Error Messages Carefully

   • Don't skip past errors or warnings — they often contain the exact solution
   • Read stack traces completely; note line numbers, file paths, error codes

2. Reproduce Consistently

   • Can you trigger it reliably? What are the exact steps? Does it happen every time?
   • If not reproducible → consult feedback-loop-construction.md to build an automated reproduction loop; don't guess
   • Record baseline: inputs, environment, version, logs, success/failure criteria

3. Check Recent Changes

   • What changed that could cause this? Git diff, recent commits, new dependencies, config changes, environmental differences

4. Gather Evidence in Multi-Component Systems

   • Instrument each component boundary: log what enters and exits
   • Run once to see where data breaks, then focus investigation there

5. Trace Data Flow (when error is deep in call stack)

   • Where does bad value originate? What called this with bad value?
   • Keep tracing up until you find the source. Fix at source, not at symptom.
   • For the complete backward tracing technique, see root-cause-tracing.md.

6. Drill Down Through Diagnostic Layers

   Start at L1. Exhaust all "why" questions at each layer before descending. The chain is open-ended — architecture is not the endpoint.

   L1 Symptom:     what failed? where? exact reproduction?
   L2 Logic:       which branch, invariant, or state transition is wrong?
   L3 System:      which component boundary, dependency, or ownership seam?
   L4 Architecture: what design choice, duplicated owner, or fallback chain?
   L5 Cross-system: which API / SLA / timing contract between systems?
   L6 Platform:    what runtime / OS / framework constraint?
   L7 Spec gap:    who never defined correct behavior for this case?
   

   Hard signal definitions (H/T/D) are in the Quality Gate — apply them there, not during initial investigation.

Phase 2: Pattern Analysis

1. Find working examples in the same codebase — what works that's similar?
2. Compare against references — read completely, don't skim
3. Identify differences between working and broken — list every difference
4. Understand dependencies — config, environment, assumptions
5. Locate the canonical owner — which file/module should own this? Multiple owners = a finding, not normality

Phase 3: Hypothesis and Testing

1. Form single hypothesis: "I think X is the root cause because Y" — be specific
2. Test minimally: smallest possible change, one variable at a time. Prefer instrumentation over code edits while still proving the cause.
3. Verify: worked? → Phase 4. Didn't? → Form NEW hypothesis. Don't stack fixes.
4. When you don't know: say "I don't understand X", don't pretend
5. Run Reflection at the end of each loop:
   • Goal | DeeperCause (yes/no/uncertain) | Evidence | Risk/Unknown | Decision (exit/iterate/escalate)
   • If DeeperCause = uncertain → continue or escalate. Only exit when root cause is deep enough and evidence is sufficient.

Phase 4: Implementation

Fix the root cause, not the symptom:

1. Create Failing Test Case

   • Simplest possible reproduction. One-off test script if no framework.
   • MUST have before fixing.

2. Implement Single Fix

   • Address the root cause identified. ONE change at a time.
   • No "while I'm here" improvements. No bundled refactoring.
   • Prefer changing the canonical owner instead of stacking more logic into a fallback path.

3. Verify Fix

   • Test passes now? No other tests broken? Issue actually resolved?
   • Verify the intended compatibility boundary still holds.
   • Verify you did not silently move authority to the wrong layer.

4. If Fix Doesn't Work

   • STOP. Count: How many fixes have you tried?
   • If < 3: Return to Phase 1, re-analyze with new information.
   • If ≥ 3: STOP and question the architecture (step 6 below). DON'T attempt Fix #4 without architectural discussion.

4bis. Post-Fix Differential Diagnosis

After applying a fix, if ANY symptom persists:

STOP. Do NOT attempt another fix without diagnosis.

1. Isolate the residual symptom precisely — what exactly remains?
2. Trace its causal chain independently (fresh Phase 1 run).
3. Compare with the causal chain of the fixed symptom:

Residual pattern	Diagnosis	Action
Same reproduction conditions as fixed symptom	Fix is incomplete	Continue drilling from same source
Different reproduction conditions, chains converge to same source	Fix was at wrong depth	Re-drill from the shared source
Different reproduction conditions, chains diverge	Compound root cause (≥2 independent roots)	Each root needs its own fix
Same symptom, reduced but not eliminated	Fix was a downstream patch	Re-drill from source

4. If uncertain whether convergent or divergent: escalate. Do not guess.

Compound root cause forms:

• True compound — ≥2 independent bugs surfaced together
• Single-root multi-symptom — 1 root, ≥2 symptom paths → fix root, all resolve
• Chain causal — A causes B causes C → fix A, B and C auto-resolve

5. If 3+ Fixes Failed: Question Architecture

   Pattern indicating architectural problem:

   • Each fix reveals new shared state/coupling/problem in different place
   • Fixes require "massive refactoring" to implement
   • Each fix creates new symptoms elsewhere

   STOP and question fundamentals. Discuss with your human partner before attempting more fixes. This is NOT a failed hypothesis — this is a wrong architecture.

6. Deliver Dual-Track Closure

   For bug fixes, refactors, contract changes, or governance cleanup, always produce:

   Repair track — root cause, canonical owner, smallest necessary change, compatibility boundary, verification method.

   Retirement track — old owner / fallback / patch, whether it is still active on the main path, the only reason to keep it (if any), trigger for deletion, verification needed before removal.

   Never add a new owner, fallback, prompt branch, or adapter path without stating what happens to the old one.

Quality Gate

Before you claim debugging is complete:

0. Workspace record for non-trivial debugging — if this is medium+ complexity or it writes docs/aegis/ records, initialize/check through the helper when available:

   python scripts/aegis-workspace.py init --root <target-project-root>
   python scripts/aegis-workspace.py new-work --root <target-project-root> ...
   python scripts/aegis-workspace.py add-evidence --root <target-project-root> --work <YYYY-MM-DD-slug> ...
   python scripts/aegis-workspace.py check --root <target-project-root>
   

   These records are method-pack evidence trails only. They do not grant authoritative completion.

1. Stop-when review — re-read the diagnostic layer where you stopped. Did you reach "no deeper why remains" or a T-class terminal boundary? If the chain ended at L1-L2 and the evidence is conclusive, that is a valid endpoint. If there are still unexplained "why" questions, continue drilling before claiming done.

2. Hard signal check — apply these countable facts, not judgments:

   Must continue drilling (H-class — ANY hit = NOT done):

   • H1 — fix added a conditional branch (if / switch / catch / try)
   • H2 — fix touched multiple sites but only 1 covered by failing test
   • H3 — fix is at consumer/caller, not canonical owner
   • H4 — same bug pattern exists elsewhere in repo (grep for it)
   • H5 — original reproduction still produces any anomaly
   • H6 — git log --grep shows this symptom was "fixed" before → Read that commit's diff. Understand why it failed. Do not repeat the same patch pattern.

   Terminal unactionable (T-class — any hit = stop drilling, switch to mitigation):

   • T1 — required change is outside this repo's boundary
   • T2 — would break published API contract with no migration path
   • T3 — root is undefined spec behavior (nobody defined correctness)
   • T4 — required permission or information is unavailable → On T-class: record root cause + system boundary + architecture review: what boundary vulnerability did this expose? can the system be made more resilient to this class of external failure?

   Depth sufficient (D-class — ALL must pass before claiming done):

   • D0 — fix eliminated ≥1 code path (paths after ≤ paths before)
   • D1 — fix eliminated ≥1 conditional branch (not added a fallback)
   • D2 — fix is at canonical owner
   • D3 — original reproduction steps no longer trigger any anomaly
   • D4 — no same-pattern occurrences remain unaddressed in repo

3. Reflection — re-run Goal / DeeperCause / Evidence / Risk/Unknown / Decision

4. Confirm the fix addressed the source, not just the sample

5. Retirement surface — did it shrink, stay, or grow?

6. Confidence:

   • A = direct evidence and regression coverage support the root-cause conclusion
   • B = strong evidence, limited coverage or some bounded unknowns remain
   • C = partial evidence only; do not present as fully resolved

If confidence is not at least B, do not speak as if the issue is fully closed.

Red Flags - STOP and Follow Process

If you catch yourself thinking:

• "Quick fix for now, investigate later"
• "Let me just try changing X and see if it works" (ignoring evidence, error messages, or hard signals)
• "Add multiple changes, run tests"
• "Skip the test, I'll manually verify"
• "I don't fully understand but this might work"
• Diagnosing by intuition — "It's probably X", listing fixes without investigation, proposing solutions before tracing data flow
• "One more fix attempt" (when already tried 2+)
• "Let's just add another fallback" instead of finding root cause
• "We can keep both owners for now" without a retirement condition
• Accepting a partial fix without differential diagnosis

ALL of these mean: STOP. Return to Phase 1.

If 3+ fixes failed: Question the architecture (see Phase 4 Step 5) If symptoms persist after fix: Run differential diagnosis (see Phase 4 Step 4bis)

Human Partner Signals

If you hear "Is that not happening?", "Will it show us...?", "Stop guessing", "Ultrathink this" → STOP. Return to Phase 1.

When Process Reveals "No Root Cause"

If investigation reveals the issue is truly environmental, timing-dependent, or external: document what you investigated, implement appropriate handling (retry, timeout, error message), add monitoring.

Supporting Techniques

See root-cause-tracing.md, defense-in-depth.md, condition-based-waiting.md, feedback-loop-construction.md in this directory for deeper guidance on specific diagnostic scenarios.