Monitoring Setup

Designs comprehensive monitoring strategies including dashboards, alerting rules, SLO definitions, and error budgets — turning raw metrics into actionable operational intelligence.

Guiding Principle

"You cannot improve what you do not measure, and you should not alert on what you cannot act upon."

Procedure

Step 1 — Define Service Level Objectives

1. Identify critical user journeys and the services that support them.
2. Define SLIs (Service Level Indicators): latency, error rate, throughput, availability.
3. Set SLO targets: e.g., "99.9% of requests complete in under 300ms over a 30-day window."
4. Calculate error budgets: (1 - SLO) * total requests = allowable failures.
5. Define error budget policies: what happens when the budget is exhausted (feature freeze, reliability sprint).

Step 2 — Design Dashboards

1. System Overview Dashboard: Service health, request rate, error rate, latency percentiles (p50, p95, p99).
2. Business Metrics Dashboard: Revenue-impacting metrics, conversion rates, feature adoption.
3. Infrastructure Dashboard: CPU, memory, disk, network, pod count, node health.
4. Deployment Dashboard: Deploy frequency, rollback count, change failure rate, lead time.
5. Follow the USE method (Utilization, Saturation, Errors) for infrastructure and RED method (Rate, Errors, Duration) for services.

Step 3 — Configure Alerting Rules

1. Alert on symptoms (error rate, latency) not causes (CPU, memory) — unless causes predict imminent failure.
2. Define severity levels: PAGE (wake someone up), TICKET (fix in business hours), LOG (informational).
3. Set appropriate thresholds with burn-rate alerts (fast burn vs. slow burn against error budget).
4. Configure alert routing: on-call rotation, escalation paths, notification channels.
5. Include runbook links in every alert so responders know what to do.

Step 4 — Operationalize and Iterate

1. Establish on-call rotation with clear responsibilities and escalation.
2. Schedule quarterly SLO reviews to adjust targets based on data.
3. Track alert noise: if an alert fires and requires no action, it should be tuned or removed.
4. Conduct post-incident reviews that feed back into monitoring improvements.
5. Document all dashboards, alerts, and SLOs in an operational catalog.

Quality Criteria

• Every SLO has a corresponding SLI, dashboard panel, and alert.
• Alerts include runbook links and are classified by severity.
• Error budgets are tracked and have defined exhaustion policies.
• Dashboard load time is under 5 seconds with 30 days of data.

Anti-Patterns

• Alerting on every metric without considering actionability (alert fatigue).
• Setting SLOs at 100% (impossible and prevents all change).
• Dashboards with 50 panels that no one reads.
• Monitoring infrastructure but not user experience.