**Date**: YYYY-MM-DD HH:MM UTC
Generates structured incident mitigation plans with three-horizon actions, risk assessment, and communication templates. Use this during active incidents to systematically restore service and prevent recurrence.
/plugin marketplace add anton-abyzov/specweave/plugin install sw-infra@specweaveDate: YYYY-MM-DD HH:MM UTC Incident: [Brief description] Root Cause: [Root cause if known, or "Under investigation"] Severity: SEV1 / SEV2 / SEV3 Created By: [Name]
Problem: [What's broken in one sentence]
Impact: [Who's affected and how]
Solution: [High-level approach]
ETA: [Estimated time to resolution]
Example:
Problem: Database connection pool exhausted due to connection leak
Impact: All users unable to access application (100% downtime)
Solution: Restart application + fix connection leak in code
ETA: 30 minutes (service restored in 5 min, permanent fix in 30 min)
Goal: Stop the bleeding, restore service immediately
Actions:
Example:
- [ ] Restart payment service to release connections
- What: Restart payment service to release database connections
- How: `systemctl restart payment-service`
- Impact: All 100 connections released, service restored
- Risk: Low (stateless service, graceful restart)
- Rollback: N/A (restart is safe)
- ETA: 2 minutes
- Owner: Jane (SRE)
- [ ] Monitor connection pool for 5 minutes
- What: Verify connections stay below 80%
- How: `watch -n 5 'psql -c "SELECT count(*) FROM pg_stat_activity"'`
- Impact: Early detection if issue recurs
- Risk: None (monitoring only)
- Rollback: N/A
- ETA: 5 minutes
- Owner: Jane (SRE)
Success Criteria:
Goal: Tactical fix to prevent immediate recurrence
Actions:
Example:
- [ ] Fix connection leak in payment service code
- What: Add `finally` block to close connection in error path
- How: Deploy hotfix branch `fix/connection-leak`
- Impact: Connections properly closed, no leak
- Risk: Medium (code change requires testing)
- Rollback: `git revert <commit>` + redeploy
- ETA: 30 minutes (test + deploy)
- Owner: Mike (Developer)
- [ ] Increase connection pool size
- What: Increase max_connections from 100 to 200
- How: ALTER SYSTEM SET max_connections = 200; SELECT pg_reload_conf();
- Impact: More headroom for traffic spikes
- Risk: Low (more connections = more memory, but server has capacity)
- Rollback: ALTER SYSTEM SET max_connections = 100; SELECT pg_reload_conf();
- ETA: 5 minutes
- Owner: Tom (DBA)
- [ ] Add connection pool monitoring alert
- What: Alert when connections >80% of max
- How: Create CloudWatch/Grafana alert
- Impact: Early warning before exhaustion
- Risk: None (monitoring only)
- Rollback: Disable alert
- ETA: 15 minutes
- Owner: Jane (SRE)
Success Criteria:
Goal: Permanent fix and prevention
Actions:
Example:
- [ ] Add automated test for connection cleanup
- What: Integration test that verifies connections are closed in error paths
- Priority: P1
- Due Date: 2025-10-27
- Owner: Lisa (QA)
- [ ] Add connection timeout configuration
- What: Set connection_timeout = 30s in database config
- Priority: P2
- Due Date: 2025-10-28
- Owner: Tom (DBA)
- [ ] Review all database queries for connection leaks
- What: Audit all DB queries to ensure proper cleanup
- Priority: P3
- Due Date: 2025-11-02
- Owner: Mike (Developer)
- [ ] Load test for high-traffic events
- What: Load test with 10x normal traffic to find bottlenecks
- Priority: P3
- Due Date: 2025-11-10
- Owner: John (DevOps)
- [ ] Update runbook with new findings
- What: Document connection leak troubleshooting steps
- Priority: P3
- Due Date: 2025-10-28
- Owner: Jane (SRE)
Success Criteria:
| Action | Risk Level | Risk Description | Mitigation |
|---|---|---|---|
| [Action 1] | Low/Med/High | [What could go wrong] | [How to reduce risk] |
Example:
| Restart service | Low | Brief downtime (5s) | Use graceful restart, off-peak time |
| Deploy code fix | Medium | Bug in fix could worsen issue | Test in staging first, have rollback ready |
| Increase connection pool | Low | More memory usage | Server has capacity, monitor memory |
| Risk | Impact | Probability |
|---|---|---|
| [Risk 1] | [Impact if we do nothing] | High/Med/Low |
Example:
| Service remains down | All users affected, revenue loss | High (will recur) |
| Connection leak worsens | Database crashes | High |
| SLA breach | Customer refunds, reputation damage | Medium |
Incident Channel: #incident-YYYYMMDD-title
Update Frequency: Every [X] minutes
Stakeholders to Notify:
Update Template:
[HH:MM] Update:
- Status: [Investigating / Mitigating / Resolved]
- Root Cause: [Known / Under investigation]
- Current Action: [What we're doing now]
- Next Steps: [What's next]
- ETA: [Estimated resolution time]
Status Page: [URL]
Update Frequency: Every [X] minutes or when status changes
Status Page Template:
[HH:MM] Investigating: We are currently investigating [issue description]. Our team is actively working on a resolution.
[HH:MM] Identified: We have identified the issue as [root cause]. We are implementing a fix. ETA: [time].
[HH:MM] Monitoring: The fix has been deployed. We are monitoring to ensure stability.
[HH:MM] Resolved: The issue has been fully resolved. All services are operating normally. We apologize for the inconvenience.
Customer Email (if needed):
Verify all of the following:
Monitor for [X] hours after declaring resolved:
Example:
- [ ] Connection pool <50% of max
- [ ] API response time <200ms (p95)
- [ ] Error rate <0.1%
- [ ] Database CPU <70%
If mitigation actions fail or make things worse:
# Rollback code deployment
git revert <commit>
npm run deploy
# Rollback database config
ALTER SYSTEM SET max_connections = 100;
SELECT pg_reload_conf();
# Verify rollback
curl http://localhost/health
Rollback if:
After incident is resolved:
Create post-mortem (within 24 hours)
Schedule post-mortem review meeting
Track action items to completion
Update runbooks based on learnings
Share learnings with organization
# Useful commands for this incident
<command1>
<command2>
<command3>
Plan Created: YYYY-MM-DD HH:MM UTC Plan Updated: YYYY-MM-DD HH:MM UTC Status: Active / Executed / Superseded
Designs feature architectures by analyzing existing codebase patterns and conventions, then providing comprehensive implementation blueprints with specific files to create/modify, component designs, data flows, and build sequences