incident-response

Incident Response

Overview

When production breaks, speed and structure matter. Follow the process — panic and ad-hoc fixes make things worse.

When to Use

• Production outage or degradation
• Customer-reported critical bugs
• Security incidents
• Writing postmortems after resolution

Incident Phases

Phase 1: Triage (first 5 minutes)

1. Assess severity — who's affected, how badly?
2. Communicate — post in incident channel: what's broken, who's looking
3. Assign roles — Incident Commander (IC), communicator, investigators

Severity	Impact	Response
SEV1	Full outage, all users	All hands, exec notification
SEV2	Major feature broken	On-call team + backup
SEV3	Minor degradation	On-call investigates

Phase 2: Mitigate (stop the bleeding)

• Rollback — if recent deploy, revert first, investigate later
• Feature flag — disable the broken feature
• Scale — if capacity issue, add resources
• Redirect — route traffic away from broken component

Goal: restore service ASAP. Root cause comes later.

Phase 3: Investigate

• Check monitoring dashboards, logs, recent deploys
• Narrow down: when did it start? What changed?
• Use systematic-debugging skill for structured investigation

Phase 4: Resolve & Verify

• Apply fix
• Monitor for 15-30 minutes
• Confirm metrics return to normal
• Update status page

Phase 5: Postmortem (within 48 hours)

## Incident Postmortem: [Title]
**Date:** YYYY-MM-DD | **Duration:** X hours | **Severity:** SEV-N

### Summary
One paragraph: what happened, impact, resolution.

### Timeline
- HH:MM — First alert
- HH:MM — IC assigned
- HH:MM — Root cause identified
- HH:MM — Fix deployed
- HH:MM — All clear

### Root Cause
What actually broke and why.

### Action Items
- [ ] [P0] Immediate fix to prevent recurrence
- [ ] [P1] Monitoring improvement
- [ ] [P2] Process improvement

Rules

• Blameless — postmortems focus on systems, not people
• Rollback first — don't debug in production if you can revert
• Communicate early and often — silence is worse than "we're investigating"
• One IC — someone owns the incident, others support

Integration

• magic-powers:systematic-debugging — structured investigation during Phase 3
• magic-powers:technical-writing — write clear postmortem
• magic-powers:infrastructure-review — prevent recurrence