Chaos Engineering Skill

Load tests tell you if your service handles traffic. Chaos Engineering tells you if it survives reality — network partitions, disk failures, dependency outages, and clock skew.

When to Activate

• Designing a chaos experiment for a service
• Running a GameDay or resilience review
• Validating that circuit breakers, retries, and fallbacks actually work
• Moving from "we think we're resilient" to "we have evidence"
• Setting up continuous chaos in CI/CD
• Verifying that a new dependency (external API, database, message queue) has correct timeout and fallback handling before go-live
• Preparing a team for an on-call rotation by building evidence that the service degrades gracefully under realistic failure scenarios
• Simulating a database latency spike with Toxiproxy to confirm connection pool exhaustion behavior and alert thresholds

---

Core Philosophy

Steady-State Hypothesis first. Before injecting chaos, define what "normal" looks like:

Hypothesis: Under normal conditions, the checkout service processes
≥99% of requests in <500ms and the error rate is <0.1%.

Chaos engineering is NOT about breaking things randomly. It's about:

1. Defining expected behavior (steady state)
2. Hypothesizing that behavior holds under adverse conditions
3. Injecting a controlled fault
4. Observing whether the hypothesis holds
5. Learning and fixing when it doesn't

---

Blast Radius Minimization

Always start small:

Scope	Example	When
Single instance	One pod, one container	First experiment
Canary traffic	5% of requests	After single instance success
Single AZ	One availability zone	Proven patterns
Full service	All instances	Only with proven fallback
Multi-service	Cascading failure	Advanced teams only

Abort criteria: Define rollback triggers before starting. If error rate exceeds X% or latency exceeds Yms — stop and restore.

---

Experiment Types

Network Faults

Simulate the most common production failure: network degradation.

# Linux — tc netem (no external tool needed)
# Add 100ms latency to all outbound traffic from eth0
tc qdisc add dev eth0 root netem delay 100ms

# Add 100ms latency with 30ms jitter (more realistic)
tc qdisc add dev eth0 root netem delay 100ms 30ms

# Simulate 10% packet loss
tc qdisc add dev eth0 root netem loss 10%

# Remove (cleanup)
tc qdisc del dev eth0 root

# macOS — comcast (wraps pfctl)
brew install comcast
comcast --device=en0 --latency=100 --target-bw=1000
comcast --device=en0 --stop

Toxiproxy — Network Simulation Proxy (recommended for services)

# Install
brew install toxiproxy
# or: docker run -d -p 8474:8474 -p 5432:5432 ghcr.io/shopify/toxiproxy

# Create a proxy for your database
toxiproxy-cli create postgres_proxy -l localhost:25432 -u your-db-host:5432

# Add latency toxic
toxiproxy-cli toxic add postgres_proxy -t latency -a latency=500

# Add connection limiter (simulate connection pool exhaustion)
toxiproxy-cli toxic add postgres_proxy -t limit_data -a bytes=0

# Simulate connection reset
toxiproxy-cli toxic add postgres_proxy -t reset_peer

# Remove toxic (restore)
toxiproxy-cli toxic remove postgres_proxy -n latency_1

CPU / Memory Pressure

# CPU stress (Linux/macOS) — stress-ng
stress-ng --cpu 4 --timeout 60s

# Memory pressure
stress-ng --vm 2 --vm-bytes 512M --timeout 60s

# Go: simulate CPU spike in test
go test -bench=. -benchtime=30s -cpuprofile=cpu.out

Pod Killing (Kubernetes)

# Manual: kill random pods in a deployment
kubectl delete pod -l app=my-service --grace-period=0 -n production

# Chaos Mesh: declarative pod failure
cat <<EOF | kubectl apply -f -
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: pod-failure-example
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces: [production]
    labelSelectors:
      app: my-service
  scheduler:
    cron: "@every 5m"
EOF

---

Tools

Toxiproxy (Network Simulation — Development/Staging)

• Best for: service-to-service network faults in dev/staging
• Supports: latency, bandwidth, packet loss, connection timeouts, connection resets
• Language-agnostic: configure via CLI or REST API

Chaos Mesh (Kubernetes)

• Best for: Kubernetes-native chaos (pod kill, network chaos, IO chaos, time skew)
• Install: helm install chaos-mesh chaos-mesh/chaos-mesh -n chaos-testing
• Features: CRD-based, fine-grained selectors, schedules, workflows

Litmus (Kubernetes — CNCF project)

• Best for: CNCF-aligned teams, chaos workflows with validation
• Hub of pre-built experiments: https://hub.litmuschaos.io

Gremlin (SaaS — Enterprise)

• Best for: production chaos with fine-grained controls and rollback
• Requires agent installation, has free tier
• Supports: CPU, memory, disk, network, shutdown, time travel, certificates

AWS Fault Injection Simulator (AWS)

• Native to AWS, supports EC2, ECS, EKS, RDS, Lambda
• IAM-based access control, built-in safety guardrails

---

GameDay Format

Before (1 week prior)

1. Define scope: Which service, which failure mode
2. Write hypothesis: Specific, measurable steady-state
3. Set abort criteria: Error rate threshold, latency threshold, blast radius limit
4. Prepare monitoring: Dashboards open, alerts set
5. Notify stakeholders: On-call, service owners

During

# Chaos Experiment Protocol — [Date]

## Service: [name]
## Hypothesis: [specific statement]
## Steady State: [measurable: p99 <Xms, error rate <Y%]

## Fault to Inject
- Type: [network latency / pod kill / CPU pressure]
- Scope: [single instance / canary 5%]
- Duration: [10 minutes]
- Tool: [Toxiproxy / Chaos Mesh / manual tc]

## Abort Criteria
- Error rate exceeds [X%]
- Latency p99 exceeds [Yms]
- On-call pages triggered

## Timeline
[HH:MM] Baseline confirmed
[HH:MM] Fault injected
[HH:MM] Observation period
[HH:MM] Fault removed
[HH:MM] Recovery confirmed

## Findings
- Hypothesis: [CONFIRMED / REJECTED]
- Observed behavior: [...]
- Action items: [...]

After

• Write up findings in 48h
• File action items as tickets
• Update runbooks if new failure modes discovered
• Schedule follow-up experiment after fixes

---

Maturity Model

Level	Characteristics
0 — Unprepared	No chaos testing. "It'll be fine."
1 — Manual	Occasional GameDays, no tooling, learning phase
2 — Structured	Regular GameDays, hypothesis-driven, tooling (Toxiproxy/Litmus)
3 — Automated	Chaos in staging CI, experiments in pre-prod before every release
4 — Continuous	Chaos in production, automated rollback, SLO-gated experiments

Start at Level 1. Most teams benefit most from Level 2–3.

---

First Experiment Checklist

• Service has health endpoint and SLI dashboards
• On-call aware and available during experiment
• Abort criteria defined before starting
• Experiment in staging, not production (first time)
• Fault injection tool tested in isolation first
• Recovery procedure documented
• Post-experiment retrospective scheduled

---

Reference

• Principles of Chaos Engineering: https://principlesofchaos.org
• Chaos Mesh: https://chaos-mesh.org
• Litmus Hub: https://hub.litmuschaos.io
• Gremlin: https://www.gremlin.com
• Toxiproxy: https://github.com/Shopify/toxiproxy