Systematic Debugging

Fix bugs through evidence, not guesswork.

Core principle: ALWAYS find root cause before attempting fixes. Symptom fixes are failure.

The Iron Law

NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST

If you haven't completed Phase 1, you cannot propose fixes.

Phase 0: Quick Wins (Before Full Investigation)

Phase 0 applies to ALL unexpected failures — broken builds, failed installs, CI errors, environment issues — not just code bugs. Any non-code operational failure triggers this skill starting at Phase 0.

Before committing to the full 5-phase process, check these common causes first — they resolve ~30% of bugs in under a minute:

1. Environment — correct runtime version? Right environment variables set?
2. Dependencies — dependency install run after recent changes? Lock file in sync?
3. Configuration — correct config file loaded? Database URL, API endpoint, port correct?
4. Cache/state — stale build artifacts? Clear build cache and dependency caches or equivalent
5. Obvious typo — check the exact error location for misspellings, wrong variable names

If one of these fixes it → verify with tests → done. If not → proceed to Phase 1.

The Five Phases

You MUST complete each phase before proceeding to the next.

Phase 1: Root Cause Investigation

BEFORE attempting ANY fix:

1. Read Error Messages Carefully

   • Full stack trace, not just the message
   • Note line numbers, file paths, error codes
   • Error messages often contain the exact solution

2. Reproduce Consistently

   • Find exact steps to trigger the bug
   • If intermittent: run multiple times, note frequency, check for race conditions
   • If can't reproduce: gather more context (environment, inputs, timing)

3. Check Recent Changes

   • git log --oneline -20 — what changed recently?
   • git blame <file> — who changed the failing code and when?
   • New dependencies, config changes, environmental differences?

4. Gather Evidence in Multi-Component Systems

   When the system has multiple layers (CI → build → deploy, API → service → database):

   Add diagnostic instrumentation at each boundary BEFORE fixing:

   For EACH component boundary:
     - Log what data enters the component
     - Log what data exits the component
     - Verify environment/config propagation
     - Check state at each layer
   
   Run once → evidence shows WHERE it breaks
   THEN investigate that specific component
   

   WARNING — Instrumentation safety rules:

   • Never log auth headers, tokens, passwords, or environment variables containing secrets
   • All diagnostic instrumentation is temporary — REMOVE before committing
   • Use generic placeholders in log output (e.g., "Auth: [PRESENT]" not the actual token)

   Example (3-layer system, adapt to your shell):

   # Layer 1: API endpoint
   log "=== Request received: $METHOD $PATH ==="
   log "=== Auth header: [PRESENT/ABSENT] ==="
   
   # Layer 2: Service layer
   log "=== Service input: $INPUT ==="
   log "=== Database query: $QUERY ==="
   
   # Layer 3: Database
   log "=== Query result rows: $ROW_COUNT ==="
   

   This reveals which layer fails (API → service ✓, service → database ✗).

5. Trace Data Flow

   • Where does the bad value originate?
   • What called this function with the bad value?
   • Keep tracing up until you find the source
   • Fix at source, not at symptom

Phase 2: Pattern Analysis

1. Find working examples — locate similar working code in same codebase
2. Compare against references — if implementing a pattern, read reference implementation COMPLETELY (don't skim)
3. Identify differences — list every difference between working and broken, however small
4. Understand dependencies — what other components, settings, config, environment does this need?

Phase 3: Hypothesis and Testing

1. Form ONE hypothesis — "I think X is the root cause because Y" (specific, testable)
2. Test minimally — smallest possible change to confirm/deny, one variable at a time
3. Evaluate:
   • Confirmed → Phase 4
   • Denied → back to Phase 1 with new evidence (don't guess again)

Phase 4: Fix and Verify

1. Write failing test — simplest possible reproduction, automated if possible
2. Implement single fix — address root cause, ONE change, no "while I'm here" improvements
3. Verify fix:
   • Run reproduction steps — bug must be gone
   • Run full test suite — no new failures
   • Check related functionality — no side effects
4. Verify regression test (red-green):
   • Revert fix → test must FAIL
   • Restore fix → test must PASS

Phase 5: Escalation — When 3+ Fixes Fail, Question Architecture

If you've tried 3 or more fixes and the bug persists:

This is NOT a failed hypothesis — this is a wrong architecture.

Pattern indicating architectural problem:

• Each fix reveals new shared state, coupling, or problems in different places
• Fixes require "massive refactoring" to implement
• Each fix creates new symptoms elsewhere

STOP and question fundamentals:

• Is this pattern fundamentally sound?
• Are we continuing through sheer inertia?
• Should we redesign this component instead of patching it?

Escalate to the user. Do NOT attempt fix #4 without discussing the architecture. Autonomous default: If running non-interactively, revert to last clean state, document all attempts and the architectural issue in SESSION.md and LESSONS_LEARNED.md, mark the bug as BLOCKED, and move to the next task.

Red Flags — STOP and Return to Phase 1

Anti-Pattern	Instead
"Quick fix for now, investigate later"	Investigate NOW. Quick fixes become permanent.
"Just try changing X and see"	Form a hypothesis first
"I think it might be..." without evidence	Show the evidence
"One more fix attempt" (after 2+ failures)	3 strikes = architecture problem. Escalate.
Fixing symptoms (suppress the error)	Find and fix the root cause
Multiple changes at once	One variable at a time
"It works on my machine"	Check environment differences
Assuming the fix works	Run reproduction steps again
Skipping regression test	If it broke once, it will break again

When Investigation Reveals No Root Cause

If systematic investigation shows the issue is truly environmental, timing-dependent, or external:

1. You've completed the process — document what you investigated
2. Implement appropriate handling (retry, timeout, error message)
3. Add monitoring/logging for future investigation

But: 95% of "no root cause" cases are incomplete investigation.

Debugging Tools

Use the right tool for the situation:

• Log statements — quick, low-risk, good for tracing flow
• Debugger/breakpoints — inspect state at a specific point
• git bisect — find which commit introduced the bug
• git blame — understand why code was written a certain way
• Minimal reproduction — strip away everything unrelated

Transitions

After fix verified: Invoke mbscode:verification-before-completion for full test suite confirmation. If bug reveals design issue: Invoke mbscode:brainstorming to redesign the affected component. Note: this creates a new design cycle — the original debugging task is complete once the fix is verified.