monitoring-ops

Monitoring Operations

Comprehensive observability patterns covering the three pillars (metrics, logging, tracing), alerting strategies, dashboard design, and infrastructure monitoring for production systems.

---

Three Pillars Quick Reference

Use this table to decide which observability signal fits your need:

Pillar	Best For	Tools	Data Type
Metrics	Aggregated numeric measurements, trends, alerting on thresholds	Prometheus, Datadog, CloudWatch, StatsD	Time-series (numeric)
Logs	Discrete events, error details, audit trails, debugging context	Loki, ELK, CloudWatch Logs, Fluentd	Unstructured/structured text
Traces	Request flow across services, latency breakdown, dependency mapping	Jaeger, Tempo, Zipkin, Datadog APM	Span trees (structured)

When to use which:

• "How many requests per second?" → Metrics (counter + rate)
• "Why did this specific request fail?" → Logs (error message + stack trace)
• "Where is the latency in this request?" → Traces (span waterfall)
• "Is the system healthy right now?" → Metrics (gauges + alerts)
• "What happened at 3:42 AM?" → Logs (timestamped event search)
• "Which downstream service caused the timeout?" → Traces (span analysis)

Correlation is key: Connect all three by embedding trace_id in log entries, recording exemplars in metrics, and linking trace spans to log queries.

---

Metrics Type Decision Tree

Use this tree to select the correct metric type:

What are you measuring?
│
├─ A count of events that only goes up?
│  └─ COUNTER
│     Examples: http_requests_total, errors_total, bytes_sent_total
│     Use rate() or increase() to get per-second or per-interval values
│     Never use a counter's raw value — it resets on restart
│
├─ A current value that goes up AND down?
│  └─ GAUGE
│     Examples: temperature_celsius, active_connections, queue_depth
│     Use for snapshots of current state
│     Can use avg_over_time(), max_over_time() for trends
│
├─ A distribution of values (latency, size)?
│  │
│  ├─ Need aggregatable quantiles across instances?
│  │  └─ HISTOGRAM
│  │     Examples: http_request_duration_seconds, response_size_bytes
│  │     Define buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10]
│  │     Use histogram_quantile() for percentiles (p50, p95, p99)
│  │     Aggregatable across instances (histograms can be summed)
│  │
│  └─ Need pre-calculated quantiles on a single instance?
│     └─ SUMMARY
│        Examples: go_gc_duration_seconds
│        Pre-calculates quantiles client-side
│        NOT aggregatable across instances
│        Prefer histogram unless you have a specific reason
│
└─ None of the above?
   └─ INFO metric (labels only, value=1)
      Examples: build_info{version="1.2.3", commit="abc123"}
      Use for metadata exposed as metrics

Rule of thumb: Start with counters and histograms. Add gauges for current state. Avoid summaries unless you have a compelling reason.

---

Alerting Decision Tree

What type of alert do you need?
│
├─ Known threshold with a fixed boundary?
│  └─ THRESHOLD-BASED
│     Example: CPU > 90% for 5 minutes
│     Pros: Simple, predictable, easy to understand
│     Cons: Requires manual tuning, doesn't adapt to patterns
│     Best for: Resource limits, error rate spikes, queue depth
│
├─ Normal behavior varies by time/season?
│  └─ ANOMALY-BASED
│     Example: Traffic 3 standard deviations below normal for this hour
│     Pros: Adapts to patterns, catches novel failures
│     Cons: Noisy during transitions, requires training data
│     Best for: Traffic patterns, business metrics, gradual degradation
│
└─ Defined reliability targets?
   └─ SLO-BASED (PREFERRED)
      Example: Error budget burn rate > 14.4x for 1 hour
      Pros: Aligned with user impact, reduces noise, principled
      Cons: Requires SLI/SLO definition, more complex setup
      Best for: User-facing services, platform reliability

Severity Levels

Severity	Response	Examples	Routing
Critical (P1)	Page on-call immediately	Service down, data loss risk, security breach	PagerDuty high-urgency, phone call
Warning (P2)	Investigate within hours	Elevated error rate, disk 80% full, SLO burn rate elevated	PagerDuty low-urgency, Slack alert channel
Info (P3)	Review next business day	Deployment completed, certificate expiring in 30 days	Slack info channel, ticket auto-created

When to Page vs When to Ticket

Page (wake someone up) when:

• Users are currently impacted
• Data loss is occurring or imminent
• Security incident is active
• Error budget will be exhausted within hours

Create ticket (don't page) when:

• Issue is not user-facing yet
• Automated remediation is possible
• Degradation is slow and has runway
• Issue is during business hours and can be triaged normally

---

Structured Logging Quick Reference

Standard JSON Log Format

{
  "timestamp": "2026-03-09T14:32:01.123Z",
  "level": "ERROR",
  "message": "Failed to process payment",
  "service": "payment-api",
  "version": "1.4.2",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "request_id": "req-abc123",
  "user_id": "usr-789",
  "error": {
    "type": "PaymentGatewayTimeout",
    "message": "Gateway response timeout after 30s",
    "stack": "..."
  },
  "duration_ms": 30042,
  "http": {
    "method": "POST",
    "path": "/api/v1/payments",
    "status_code": 504
  }
}

Log Level Decision Guide

Level	When to Use	Examples
DEBUG	Development only, verbose internal state	Variable values, SQL queries, cache hits/misses
INFO	Normal operations worth recording	Request completed, job started/finished, config loaded
WARN	Degraded but still functioning	Retry succeeded, fallback used, approaching limit
ERROR	Operation failed, needs attention	Payment failed, API call error, constraint violation
FATAL	Process cannot continue, must exit	Database unreachable at startup, invalid config, OOM

Rules:

• Never log at ERROR for expected conditions (user input validation → WARN)
• Every ERROR should be actionable — if no one will act on it, use WARN
• DEBUG should be off in production by default
• INFO should not be noisy — 1-5 log lines per request, not 50

Correlation IDs

• Generate a request_id (UUID v4 or ULID) at the edge/gateway
• Propagate through all internal services via headers (X-Request-ID)
• Include trace_id and span_id from distributed tracing
• Log all three IDs in every log entry for cross-referencing

---

Distributed Tracing Quick Reference

Core Concepts

• Trace: End-to-end journey of a request across all services
• Span: A single unit of work (HTTP call, DB query, function execution)
• Context propagation: Passing trace/span IDs between services via headers

W3C TraceContext Header

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
              │  │                                  │                  │
              │  │                                  │                  └─ flags (01=sampled)
              │  │                                  └─ parent span ID (16 hex)
              │  └─ trace ID (32 hex)
              └─ version (00)

Sampling Strategies

Strategy	How It Works	Use When
Head-based (ratio)	Decide at trace start, propagate decision	Low traffic, need predictable volume
Always-on	Sample everything	Development, low-traffic services
Parent-based	Follow parent's sampling decision	Default for most services
Tail-based	Decide after trace completes (at Collector)	Need error/slow traces, high traffic

Recommendation: Use parent-based + tail-based at the Collector. This captures all error traces and slow traces while controlling volume.

Trace ID in Logs

Always include trace_id in structured log entries. This enables jumping from a log line to the full trace view:

Log entry → trace_id → Jaeger/Tempo → full request waterfall

---

Tool Selection Matrix

Feature	Prometheus + Grafana	Datadog	Grafana Cloud	CloudWatch
Cost	Free (infra costs)	$$$$ (per host/metric)	$$ (usage-based)	$$ (AWS-native)
Setup complexity	High (self-managed)	Low (SaaS agent)	Medium (managed)	Low (AWS-native)
Metrics	Prometheus (excellent)	Built-in (excellent)	Mimir (excellent)	Built-in (good)
Logs	Loki (good)	Built-in (excellent)	Loki (good)	CloudWatch Logs (good)
Traces	Jaeger/Tempo (good)	APM (excellent)	Tempo (good)	X-Ray (adequate)
Alerting	Alertmanager (good)	Built-in (excellent)	Grafana Alerting (good)	CloudWatch Alarms (adequate)
Dashboards	Grafana (excellent)	Built-in (excellent)	Grafana (excellent)	Dashboards (adequate)
Retention	Configurable (unlimited)	15 months default	Configurable	Up to 15 months
Multi-cloud	Yes	Yes	Yes	AWS only
Best for	Cost-conscious, control	Full-featured, enterprise	Open-source + managed	AWS-native shops

Recommendation path:

• Starting out / budget-conscious: Prometheus + Grafana + Loki + Tempo (all free, self-hosted)
• Small team, want managed: Grafana Cloud free tier (10k metrics, 50GB logs, 50GB traces)
• Enterprise, need everything: Datadog (expensive but comprehensive)
• AWS-only shop: CloudWatch + X-Ray (simplest if already on AWS)

---

Dashboard Design

USE Method (Infrastructure)

For every resource (CPU, memory, disk, network):

Signal	Question	Metric Example
Utilization	How busy is it?	node_cpu_seconds_total (% busy)
Saturation	How overloaded is it?	node_load1 (run queue length)
Errors	Are there error events?	node_network_receive_errs_total

RED Method (Services)

For every service endpoint:

Signal	Question	Metric Example
Rate	How many requests per second?	rate(http_requests_total[5m])
Errors	How many are failing?	rate(http_requests_total{status=~"5.."}[5m])
Duration	How long do they take?	histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

Four Golden Signals (Google SRE)

Signal	What to Measure	Alert Threshold Guidance
Latency	Time to serve a request (distinguish success vs error latency)	p99 > 2x baseline
Traffic	Demand on the system (requests/sec, sessions, transactions)	Anomaly detection
Errors	Rate of failed requests (explicit 5xx, implicit policy violations)	> 0.1% of traffic
Saturation	How "full" the service is (CPU, memory, queue depth)	> 80% capacity

Dashboard Layout Best Practices

1. Top row: Key health indicators (error rate, latency p99, availability %)
2. Second row: Traffic and throughput (requests/sec, active users)
3. Third row: Resource utilization (CPU, memory, disk, network)
4. Bottom rows: Detailed breakdowns (by endpoint, by status code, by region)
5. Use variables: Service, environment, time range as dropdown selectors
6. Include annotations: Deployments, incidents, config changes as vertical markers

---

Common Gotchas

Gotcha	Why It Happens	Fix
Cardinality explosion	Using unbounded label values (user ID, request path, query string)	Use bounded labels only; aggregate high-cardinality data in logs, not metrics
Alert fatigue	Too many alerts, too sensitive thresholds, alerts on non-actionable symptoms	Require runbook for every alert; tune thresholds; use SLO-based alerting
Missing correlation IDs	Logs, metrics, and traces not linked together	Include trace_id in all log entries; use exemplars in metrics
Sampling bias	Head-based sampling drops error/slow traces at high sample rates	Use tail-based sampling at the Collector to always capture errors and slow traces
Log volume costs	DEBUG or verbose INFO in production, logging full request/response bodies	Set production to INFO minimum; truncate large payloads; use sampling for verbose paths
Metric naming inconsistency	Different teams use different naming conventions	Adopt OpenMetrics naming: namespace_subsystem_unit_suffix (e.g., http_server_request_duration_seconds)
Dashboard sprawl	Everyone creates dashboards, nobody maintains them	Standardize with USE/RED templates; review quarterly; delete unused dashboards
SLO too aggressive	Setting 99.99% availability without the budget or architecture for it	Start with 99.5% or 99.9%; tighten only when consistently meeting targets with margin
Missing baseline	Alerting on absolute thresholds without understanding normal behavior	Collect 2-4 weeks of baseline data before setting alert thresholds
Over-instrumentation	Instrumenting every function, creating too many spans/metrics	Instrument at service boundaries; use auto-instrumentation for HTTP/DB/gRPC; add manual spans selectively
Ignoring metric staleness	Assuming a metric that stops reporting means zero	Use absent() or up == 0 to detect missing scrapers; distinguish "zero" from "not reporting"
Alerting on cause not symptom	Alerting on CPU usage instead of user-facing error rate	Alert on symptoms (error rate, latency); use cause metrics (CPU, memory) for investigation
No retention policy	Storing all metrics/logs at full resolution forever	Define retention tiers: 15s resolution for 2 weeks, 1m for 3 months, 5m for 1 year
Dashboard without context	Graphs with no units, no description, no threshold lines	Add units to Y-axis, threshold lines for SLOs, panel descriptions explaining what "good" looks like

---

Reference Files

File	Contents	Lines
metrics-alerting.md	Prometheus, Grafana, OpenTelemetry metrics, SLI/SLO/SLA, alert routing, runbooks, uptime monitoring	~650
logging.md	Structured logging, log levels, correlation IDs, aggregation (Loki, ELK), retention, PII masking, language-specific	~550
tracing.md	OpenTelemetry, spans, context propagation, sampling, Jaeger, async tracing, DB/HTTP/gRPC instrumentation	~600
infrastructure.md	Health checks, K8s probes, Docker HEALTHCHECK, infra metrics, APM, cost optimization, incident response	~550

---

See Also

• docker-ops — Container monitoring with cAdvisor, Docker stats, and health checks
• ci-cd-ops — Pipeline observability, deployment tracking, build metrics
• nginx-ops — Nginx access/error log parsing, request metrics, upstream monitoring
• python-observability-ops — Python-specific instrumentation with structlog, opentelemetry-python
• OpenTelemetry documentation
• Prometheus best practices
• Google SRE Book — Monitoring chapter
• Grafana dashboards library