exhaustive-systems-analysis

Exhaustive Systems Analysis

Systematic audit methodology for rooting out latent issues in codebases, particularly agent-written code that needs verification before production use.

Core Principles

1. Subsystem isolation — Analyze each subsystem separately to prevent context pollution
2. Evidence-based findings — Every issue must cite specific code locations
3. Severity-driven prioritization — Critical issues first, cosmetic issues last
4. Assume all issues will be fixed — Don't hedge; be direct about what's wrong

Workflow

Phase 1: System Decomposition

Before analysis, map the system's subsystems. Auto-discover by:

1. Read project structure — Identify major modules, packages, or directories
2. Trace data flow — Follow how data enters, transforms, and exits
3. Identify side effects — File I/O, network, database, IPC, state mutations
4. Map dependencies — Which subsystems depend on which

Output a subsystem table:

| # | Subsystem | Files | Side Effects | Priority |
|---|-----------|-------|--------------|----------|
| 1 | Lock System | lock.rs | FS: mkdir, rm | High |
| 2 | API Layer | api/*.rs | Network, DB | High |
| 3 | Config Parser | config.rs | FS: read | Medium |

Priority heuristics:

• High: Side effects, state management, security, concurrency
• Medium: Business logic, data transformation, validation
• Low: Pure utilities, formatting, logging

Phase 2: Sequential Analysis

Analyze subsystems in priority order. For large codebases (>5 subsystems or >3000 LOC per subsystem), prefer clearing context between subsystems to prevent analysis drift.

For each subsystem, apply the appropriate checklist based on subsystem type.

Phase 3: Consolidation

After all subsystems analyzed:

1. Deduplicate cross-cutting findings
2. Rank all issues by severity
3. Produce final report with recommended action order

---

Analysis Checklists

Select checklist based on subsystem characteristics. Apply multiple if applicable.

Stateful Systems (files, databases, caches, locks)

Check	Question
Correctness	Does code do what documentation claims?
Atomicity	Can partial writes corrupt state?
Race conditions	Can concurrent access cause inconsistency?
Cleanup	Are resources released on all exit paths (success, error, panic)?
Error recovery	Do failures leave the system in a valid state?
Stale documentation	Do comments match actual behavior?
Dead code	Are there unused code paths that could confuse maintainers?

APIs & Network (HTTP, gRPC, WebSocket, IPC)

Check	Question
Input validation	Are all inputs validated before use?
Error responses	Do errors leak internal details?
Timeout handling	Are network operations bounded?
Retry safety	Are operations idempotent or properly guarded?
Authentication	Are auth checks applied consistently?
Rate limiting	Can the API be abused?
Serialization	Can malformed payloads cause panics?

Concurrency (threads, async, channels, locks)

Check	Question
Deadlock potential	Can lock acquisition order cause deadlock?
Data races	Is shared mutable state properly synchronized?
Starvation	Can any task be indefinitely blocked?
Cancellation	Are cancellation/shutdown paths clean?
Resource leaks	Are spawned tasks/threads joined or detached properly?
Panic propagation	Do panics in tasks crash the whole system?

UI & Presentation (views, components, templates)

Check	Question
State consistency	Can UI show stale or inconsistent state?
Error states	Are all error conditions rendered appropriately?
Loading states	Are async operations properly indicated?
Accessibility	Are interactions keyboard/screen-reader accessible?
Memory leaks	Are subscriptions/observers cleaned up?
Re-render efficiency	Are unnecessary re-renders avoided?

Data Processing (parsers, transformers, validators)

Check	Question
Edge cases	Are empty, null, and boundary values handled?
Type coercion	Are implicit conversions safe?
Overflow/underflow	Are numeric operations bounded?
Encoding	Is text encoding handled consistently (UTF-8)?
Injection	Can untrusted input escape its context?
Invariants	Are data invariants enforced and documented?

Configuration & Setup (config files, environment, initialization)

Check	Question
Defaults	Are defaults safe and documented?
Validation	Are invalid configs rejected early with clear errors?
Secrets	Are secrets handled securely (not logged, not in VCS)?
Hot reload	If supported, is reload atomic and safe?
Compatibility	Are breaking changes versioned or migrated?

---

Severity Classification

Classify every finding. Assume user will fix all issues soon.

Severity	Criteria	Examples
Critical	Data loss, security vulnerability, crash in production	Unhandled panic, SQL injection, file corruption
High	Incorrect behavior users will notice	Wrong calculation, race causing wrong UI state, timeout too short
Medium	Technical debt that causes confusion or future bugs	Stale docs, misleading names, redundant code paths
Low	Cosmetic or minor improvements	Unused parameter, suboptimal algorithm (works correctly)

---

Finding Format

Every finding must follow this structure:

### [SUBSYSTEM] Finding N: Brief Title

**Severity:** Critical | High | Medium | Low
**Type:** Bug | Race condition | Security | Stale docs | Dead code | Design flaw
**Location:** `file.rs:line_range` or `file.rs:function_name`

**Problem:**
What's wrong and why it matters. Be specific.

**Evidence:**
Code snippet or reasoning demonstrating the issue.

**Recommendation:**
Specific fix. Include code if helpful.

---

Output Structure

Adapt output to project organization. Common patterns:

Pattern A: Audit Directory (recommended for 5+ subsystems)

.claude/docs/audit/
├── 00-analysis-plan.md      # Subsystem table, priorities, methodology
├── 01-subsystem-name.md     # Individual analysis
├── 02-another-subsystem.md
└── SUMMARY.md               # Consolidated findings, action items

Pattern B: Single Document (for smaller systems)

.claude/docs/audit/system-name-audit.md
# Contains: plan, all findings, summary

Pattern C: Inline with Existing Docs

If project has existing docs/ or similar, place audit artifacts there.

Always create a summary with:

• Total findings by severity
• Top 5 most critical issues
• Recommended fix order

---

Session Management

For thorough analysis:

• Small systems (<1000 LOC, <3 subsystems): Single session acceptable
• Medium systems (1000-5000 LOC, 3-7 subsystems): Clear context between phases
• Large systems (>5000 LOC, >7 subsystems): Separate sessions per subsystem

When clearing context, document progress in the analysis plan file so the next session can continue.

---

Pre-Analysis: Known Issues Sweep

Before deep analysis, scan for documented issues:

1. Check CLAUDE.md / README for "gotchas" or "known issues"
2. Search for TODO/FIXME/HACK comments
3. Review recent commits for bug fixes (may indicate fragile areas)
4. Check issue tracker if accessible

Add these as starting hypotheses—verify or refute during analysis.

---

Anti-Patterns to Avoid

Anti-Pattern	Why It's Bad	Instead
Skimming code	Misses subtle bugs	Read every line in scope
Assuming correctness	Agent code often has edge case bugs	Verify each code path
Vague findings	"This looks wrong" isn't actionable	Cite specific lines, explain why
Over-scoping	Analysis paralysis	Strict subsystem boundaries
Ignoring tests	Tests reveal assumptions	Read tests to understand intent

---

Completion Criteria

Analysis is complete when:

1. ✅ All high-priority subsystems analyzed
2. ✅ Every finding has severity, location, and recommendation
3. ✅ Summary document exists with prioritized action items
4. ✅ No "TBD" or "needs investigation" items remain
5. ✅ Cross-references between related findings added