observability-setup

Observability Setup

Ortak Doktrin

agents/shared/severity-rubric.md ve agents/shared/escalation-matrix.md default-load sayılır (agents/coordination.md §11). Bu skill'in çıktısı Critical / High / Medium / Low + kanıt formatında olmak zorunda — spekülatif Critical yasak. Sahiplik dışı bulgu ilgili agent'a delege; karar yetkisi eşiği aşılırsa kullanıcı onayı zorunlu.

Ne Zaman Kullanılır

• Yeni servis prod'a çıkacak (observability gate)
• Mevcut servis "yeşil ama incident var" — boş panel + sahte alert
• Alert fatigue / pager noise (SNR düşük)
• SLO definition/review
• Trace kopuk (cross-service context propagation)
• Migrating monitoring stack (Datadog → Prometheus, vb.)

Workflow

1) Servis envanteri

• Hangi endpoint'ler / operasyonlar?
• Critical user journey'ler (login, checkout, search, ...)?
• Bağımlılıklar (DB, cache, external API, queue)?
• SLA varsa müşteri taahhüdü.

2) SLI seçimi (Golden Signals)

Sinyal	SLI
Availability	2xx_3xx_4xx_count / total_count (5xx fail)
Latency	requests < 500ms / total veya p99 < 500ms
Throughput/Saturation	current_qps < capacity * 0.7
Quality	Cache hit %, freshness, accuracy (ürüne özel)

Az + anlamlı: 3-5 SLI / servis.

3) SLO tanımı

- service: api-svc
  slos:
    - name: availability
      sli: "rate(http_requests_total{code!~'5..'}[1m]) / rate(http_requests_total[1m])"
      target: 0.999
      window: 30d_rolling
    - name: latency_p99
      sli: "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))"
      target: 0.5      # 500ms
      window: 30d_rolling

Error budget hesabı:

• 99.9% / 30d → 43m 12s
• 99.95% / 30d → 21m 36s
• 99.99% / 30d → 4m 19s

Hedef ekibe gerçekçi olmalı; aspirational yerine maintained.

4) Burn rate alert (multi-window)

groups:
- name: api-svc-slo-burn
  rules:
  - alert: APIHighBurnFast
    expr: |
      (
        sum(rate(http_requests_total{service="api",code=~"5.."}[1h]))
        / sum(rate(http_requests_total{service="api"}[1h]))
      ) > 14.4 * 0.001    # 14.4x burn (= %2 budget / 1h)
      and
      (
        sum(rate(http_requests_total{service="api",code=~"5.."}[5m]))
        / sum(rate(http_requests_total{service="api"}[5m]))
      ) > 14.4 * 0.001
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "api-svc burning %2 of monthly budget per hour"
      runbook_url: "https://runbooks.example/api-svc/high-error-rate"

AND (1h + 5m) kondisyonu: false-positive azaltır.

5) Logging

# structured logger + correlation
import structlog

log = structlog.get_logger()
log.info(
    "order_created",
    order_id=order.id,
    customer_id=customer.id,   # NOT pii (mask)
    amount=order.total,
    trace_id=ctx.trace_id,     # OTel context
)

• JSON output stdout → log shipper (Loki/Datadog/CloudWatch).
• PII redact: email mask, IBAN trunc, password silinir (logger middleware).
• Sampling yüksek volume: head-based (1%) veya error-only full.
• Trace ↔ log correlation: trace_id her log'da; UI'da tek tıkla geçiş.
• Retention: 30d hot / 90d warm / 1y cold.

6) Tracing (OTel)

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint=OTLP_URL)))
trace.set_tracer_provider(provider)

# auto-instrument: FastAPI, requests, sqlalchemy
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
FastAPIInstrumentor.instrument_app(app)

• W3C traceparent header propagation.
• Sampling: head-based 1-10% prod; tail-based ideal (rare error full trace).
• Span: business op, DB query, external HTTP, cache lookup.
• Backend: Tempo / Jaeger / Datadog / Honeycomb.

7) Metric (RED + USE)

from prometheus_client import Counter, Histogram, Gauge

http_requests_total = Counter(
    "http_requests_total",
    "Total HTTP requests",
    labelnames=["service", "method", "endpoint", "code"],
    # NOT label: user_id (cardinality patlar)
)
http_request_duration_seconds = Histogram(
    "http_request_duration_seconds",
    "HTTP request duration",
    labelnames=["service", "method", "endpoint"],
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10),
)

• RED API/handler için.
• USE infra için (CPU/memory/disk/network).
• Cardinality kontrol: label sayısı ≤ 10, değer kombinasyon ≤ 1k.
• Counter monotonik; gauge anlık; histogram distribution.

8) Dashboard (3-tier)

Tier	İçerik	İlk panel
Overview	SLO + golden signals + deploy timeline	Error budget burn-down
Drill-down	Endpoint başına RED + slow query + cache miss	Top-5 slow endpoint
Infra	CPU/Memory/Network/Disk + DB pool + Queue depth	Saturation heatmap

• Templating: service, env, version Grafana variable.
• Annotations: deploy line, incident marker, alert fire.
• Reading order: top-left önce; saat yönünde detay.
• Heatmap distribution için (latency).

9) Synthetic check / probes

• External probe (Pingdom / Datadog Synthetics / blackbox-exporter): user-experience perspektifi, public endpoint.
• Liveness/Readiness Kubernetes:
  • Liveness: process alive (basit).
  • Readiness: traffic serve edebilir (DB connect + dependency probe).
• Startup probe (slow-init container).

10) Validation

• Alert drill: feature flag ile test alert tetikle, runbook çalıştı mı?
• Dashboard usability: yeni geliştirici 30 sn'de servis sağlığını okuyabiliyor mu?
• PII redact test: log'da gerçek email / token görünüyor mu?
• Trace propagation test: cross-service request trace ID korunuyor mu?
• SLO calculation test: budget hesabı 30d window doğru mu?

Checklist

• 3-5 SLI seçildi (golden signals)
• SLO sayı+birim+window (30d rolling tercih)
• Multi-window burn rate alert (1h+5m AND)
• Structured log JSON + trace_id + PII redact
• OTel SDK + W3C traceparent propagation
• RED metric (handler) + USE metric (infra)
• Cardinality kontrol (label kombinasyon ≤ 1k)
• 3-tier dashboard (overview/drill/infra)
• Annotation: deploy + incident
• Liveness + readiness probe
• External synthetic check
• Alert runbook link zorunlu
• Alert drill yapıldı

Antipattern

• CPU%/memory% SLI — symptom değil cause.
• Calendar month error budget — rolling tercih.
• High cardinality label (userId) — metric storage patlar.
• Cause-based alert ("disk %85 dolu") — symptom-based ("write fail") tercih.
• Runbook yok alert — actionable değil.
• for: 0s rule — flapping.
• PII log'da — masked değil.
• Trace yok prod — cross-service request anlaşılmaz.
• 20+ SLO — sinyal kayıp.
• Calendar month budget — ay sonunda yenilenir, kötü insentiv.
• Aspirational SLO (99.9999%) — ekibin sürdüremediği hedef.
• Symptom + cause alert beraber — pager fatigue.

Örnek Agent Davranışı

User: /observe-bootstrap api-svc
Agent (observability-engineer):
1. Servis envanteri: 12 endpoint, ana journey checkout (POST /orders).
2. Bağımlılık: postgres-rw, redis, payment-provider, kafka.
3. SLI seçimi:
   - availability: 5xx oran < %0.1 (30d)
   - latency_p99: < 500ms (30d)
   - cache_hit_rate: > %85 (30d)
4. SLO error budget: %0.1 / 30d = 43m 12s downtime/ay.
5. Burn rate alert: 14.4x (critical), 6x (high), 3x (medium).
6. Stack tespit: Prometheus + Loki + Tempo (OTel SDK uyumlu).
7. Diff:
   - `app/observability.py` — structlog + OTel init
   - `app/middleware.py` — PII redact, trace propagation
   - `kubernetes/api-svc-rules.yaml` — burn rate Prom rule
   - `grafana/api-svc-overview.json` — overview dashboard
8. Alert drill plan: feature flag ile %5xx tetikle, runbook test et.
9. Output: 4 dosya diff + dashboard JSON + runbook link checklist.

Çıktı Formatı

# Observability Bootstrap: <service>

## SLI/SLO
| SLI | Target | Window |

## Critical / High / Medium / Low (mevcut gap)

## Diff (özet)
```yaml
# config / SDK init / alert rule

Dashboard

• overview / drill / infra

Alert

| Severity | Burn rate | Runbook |

Validation

• Alert drill, PII test, trace propagation test