Investigation Techniques

Problem Categorization

Technical Problems

Category	Sub-Category	Common Symptoms
System Failures	App crashes, memory leaks, deadlocks, data corruption	Service unavailability, resource exhaustion, integrity errors
System Failures	Hardware, network, database, security	Connectivity loss, capacity limits, access failures
Performance	Response time: slow queries, latency, algorithmic inefficiency	High p95/p99, user-reported slowness
Performance	Throughput: thread pool exhaustion, connection limits, queue backlog	Reduced capacity, growing queues
Integration	Internal: component comms, data format, version conflicts	Interface errors, serialization failures
Integration	External: third-party availability, API changes, auth failures	Timeouts, contract violations

Operational Problems

Category	Common Symptoms
Deployment: script failures, config drift, migration errors	Failed releases, environment inconsistencies
Monitoring: alerting gaps, backup failures, incident response	Missed incidents, slow recovery
Human factors: communication gaps, knowledge silos, skill gaps	Repeated mistakes, slow onboarding

Evidence Collection

Technical Evidence Sources

Logs: application (timestamp correlation) | system/infrastructure | database | network traces

Metrics: performance/resource utilization | error rates/response time trends | user behavior/transaction patterns | infrastructure health/capacity

Configuration: system/deployment settings | code changes/VCS history (git log, blame) | env vars/dependencies | security/access controls

Evidence Validation

1. Cross-reference: verify from multiple independent sources
2. Timestamp validation: confirm event sequence accuracy
3. Completeness check: identify data gaps/corruption
4. Correlation vs causation: distinguish co-occurrence from causation

Analysis Techniques

Quantitative

• Trend: time series of metrics, error pattern frequency
• Distribution: response time percentiles, error rate across components
• Pattern recognition: log anomalies, behavior patterns, error clustering

Qualitative

• Timeline reconstruction: detailed incident timeline, correlate changes with symptoms
• Process analysis: workflow disruptions, communication flow, decision chains
• Environmental: recent changes, system load, external factors, related incidents

Solution Design Patterns

Immediate Mitigations (restore service)

Quick fixes | workarounds to minimize impact | emergency procedures | monitoring enhancements

Permanent Fixes (prevent recurrence)

Architecture modifications | code quality/defensive programming | config management/environment consistency | testing/validation improvements

Early Detection (catch faster)

Leading indicators | anomaly detection/predictive alerting | automated quality gates | threshold tuning from learnings

Solution Prioritization Matrix

Priority	Criteria	Action
P0	Active incident, users impacted	Immediate mitigation, hours
P1	Root cause fix for recurring issue	Permanent fix, current sprint
P2	Prevention for potential issues	Next sprint
P3	Systemic improvement	Backlog with evidence