resilience-failure

Resilience & Failure

Design the system so that when a part breaks — and it will — the failure is contained and the user still gets a useful (if degraded) answer instead of an error page or a cascading outage. Getting this wrong is the difference between a slow dependency and a total meltdown: the most common amplifier of an outage is the system's own reaction to it (retry storms, health-check stampedes).

When to reach for this

Any design with a remote dependency, a shared resource, or an SLA. Reach here to find single points of failure, decide what each call does when its dependency is slow or down, protect a service from being overwhelmed (rate limiting), and plan how a recovered service comes back without being crushed by the backlog.

When NOT to

Don't wrap a single in-process function or a best-effort batch job in circuit breakers and bulkheads — that's machinery for cross-process/cross-network calls (YAGNI). Don't add retries to a non-idempotent write without an idempotency key first (→ api-design) — you'll duplicate side effects. The cheapest design that meets the availability target wins; chasing an extra nine you don't need costs real complexity (→ back-of-the-envelope for what a nine actually buys).

Clarify first

• Availability target — how many nines, and is it per-request or per-feature? (→ back-of-the-envelope.)
• Blast radius — if this dependency dies, must the whole request fail, or can the feature degrade or hide?
• Idempotency — is the operation safe to retry? If not, what makes it safe (key, dedup)? (→ api-design.)
• Latency budget — how long may a call wait before a timeout is better than waiting? (→ back-of-the-envelope.)
• Limit dimension & policy — rate-limit per user / IP / API key / tenant? Hard (reject) or soft (queue/shape)? Burst tolerated?

The options

Layered defenses; most real designs combine several.

• Timeout — bound every remote call. Use everywhere; an unbounded wait is the root of most cascades.
• Retry with backoff + jitter — re-attempt transient failures with growing, randomized delays. Use for idempotent calls against blips; never naked retries.
• Circuit breaker — stop calling a dependency that's failing; fail fast and probe to recover. Use when a downstream is down or slow and retries would pile on.
• Bulkhead — isolate resources (thread pools, connection pools, queues) per dependency. Use so one slow dependency can't exhaust capacity shared by others.
• Graceful degradation — fall back to a cached/stale value, partial result, default, or hidden feature. Use when a usable-but-worse answer beats an error.
• Rate limiting / load shedding — cap inbound work; reject or shape excess. Use to protect a service from overload, abuse, or a stampeding caller.
• Redundancy / failover — run N>1 of every component; promote a standby on failure. Use to remove SPOFs. (Health checks/LB failover live in load-balancing.)

Rate-limiting algorithms (token bucket, leaky bucket, fixed/sliding window) and the circuit-breaker state machine are detailed in references/deep-dive.md.

Trade-offs

Option	What it solves	What it worsens	Change it when
Timeout	Bounds blocked threads; stops one slow call hanging the caller	Too tight → false failures; too loose → cascades	Tune to the dependency's p99, not a guess
Retry + backoff + jitter	Rides out transient blips	Multiplies load; duplicates non-idempotent writes	Add jitter + cap attempts + budget; require idempotency
Circuit breaker	Fails fast, gives a sick dependency room to recover	Adds state/tuning; can trip on a blip and over-shed	Flapping → tune thresholds / half-open probe rate
Bulkhead	Contains one failure to its own pool	Lower peak utilization; more pools to size	One noisy dependency starves others
Graceful degradation	Keeps the user served when a dependency dies	Serves stale/partial; more code paths to test	Correctness must be exact → fail closed instead
Rate limiting	Protects the service; bounds cost/abuse	Rejects legitimate bursts; needs shared state at scale	Limits too strict (valid drops) or too loose (overload)
Redundancy / failover	Removes SPOFs; survives node/region loss	Cost, replication lag, failover consistency risk	Failover drops un-replicated writes → consistency-coordination

Behavior under stress

This block exists to stop the system from amplifying its own outage.

• Retry storm: a dependency slows, every caller retries, retries pile on the retries of callers upstream, and load multiplies geometrically. Mitigate: exponential backoff with jitter, a per-request retry budget (cap total attempts), and a circuit breaker so a dead dependency isn't retried at all.
• Thundering herd on recovery: a service comes back and every queued client and expired cache entry hits it at once, knocking it over again. Mitigate: half-open circuit breakers that admit a trickle, jittered client reconnect, request coalescing, and slow-start ramp. (Cache-expiry stampede is caching.)
• Health-check stampede / accidental DDoS: aggressive health checks or load-balancer probes hammer a recovering instance. Mitigate: gentle probe intervals, fail-fast readiness, and draining. (Probe mechanics → load-balancing.)
• Timeout-less cascade: one slow dependency holds threads until the pool is exhausted, and the caller now looks "down" to its callers. Mitigate: timeouts + bulkheads everywhere.
• Rate-limiter as SPOF: a shared counter store (e.g. Redis) for limits goes down. Mitigate: fail-open (allow on limiter error) for availability, or fail-closed for protection — decide deliberately.

Monitor: error rate and p99 per dependency, retry counts, circuit-breaker state transitions, pool saturation/queue depth, rate-limit rejection rate, and "time to first success" after a recovery.

How to apply

1. Clarify the inputs — pin the availability target, blast radius per dependency, idempotency, latency budget, and the rate-limit dimension/policy (the "Clarify first" list). No defense is chosen before these are answers.
2. Pick the defenses — walk the trade-off table per dependency, not globally. Every remote call gets a timeout; add retry+jitter only where idempotent; add a circuit breaker where a sick downstream would pile on; bulkhead shared pools; choose degrade vs fail closed by whether a stale answer is acceptable.
3. Set the key knobs — timeout = the dependency's measured p99; retry cap (often 2–3) plus a per-request budget and jitter; breaker open/half-open thresholds; bulkhead pool sizes; limiter rate/burst and fail-open-vs-closed.
4. Stress-test the design — replay each amplifier from "Behavior under stress" (retry storm, recovery herd, health-check stampede, timeout-less cascade, limiter-as-SPOF) and confirm a mitigation is in place for each.
5. Size with numbers — compute composed availability along the request path (series multiplies, parallel adds nines) and confirm the target is met without over-provisioning. (→ back-of-the-envelope.)
6. Pick a provider — default to the generic recipe; only read a provider file if the user named a cloud (see "Choosing a provider").

Dos and don'ts

Do

• Bound every remote call with a timeout tuned to the dependency's p99.
• Add jitter and a retry budget so re-attempts can't multiply into a storm.
• Make a degraded response explicit (stale: true) instead of a silent lie.
• Decide fail-open vs fail-closed deliberately for limiters and breakers.
• Stress-test against the amplifiers before calling the design resilient.

Don't

• Retry a non-idempotent write without an idempotency key (→ api-design).
• Wrap in-process calls in breakers/bulkheads — that's cross-network machinery.
• Chase an extra nine the SLA doesn't require; redundancy cost is non-linear.
• Let a shared limiter or counter store become an unguarded single point of failure.
• Hammer a recovering instance with aggressive health checks or full reconnects.

Numbers that matter

Tie timeouts to the dependency's measured p99, not a round guess. Cap retries (often 2–3) and apply a budget so total attempts can't explode. Each extra "nine" of availability costs disproportionately more redundancy — know what a nine actually buys before targeting it. Composed availability matters: components in series multiply (two 99.9% deps in a request path ≈ 99.8%), redundant components in parallel add nines. For all of these — latency tables, the nines table, series/parallel availability math — see back-of-the-envelope.

Interface sketch

Two contracts are load-bearing here.

• Degraded response: make "I'm degraded" explicit, not a silent lie. Return the fallback plus a signal, e.g. { "data": [...], "stale": true, "source": "cache", "as_of": "2026-05-29T10:00Z" } so callers and clients can react.
• Rate-limit response: reject with HTTP 429 Too Many Requests and standard headers — X-RateLimit-Limit, X-RateLimit-Remaining, and Retry-After (seconds) so a well-behaved client backs off instead of retrying into the wall.

Choosing a provider

Default to the generic recipe above (resilience libraries, a token-bucket/leaky- bucket limiter, health checks, N+1 redundancy). If the user names a cloud, read references/providers/<provider>.md for the managed-service mapping, quotas/limits, and provider-specific trade-offs. If no file exists for that provider, the generic recipe is the answer.

Diagram

To visualize a fallback path (gateway → timeout on primary → dashed arrow to cache/default) or a circuit-breaker state machine, use the in-plugin architecture-diagram skill; draw the degraded path as a dashed arrow and the failed dependency in the error color.

Related building blocks

• messaging-streaming — pairs with this: a queue absorbs a write spike and a dead-letter queue contains poison messages; owned-concept lives in it for delivery guarantees and DLQ mechanics.
• load-balancing — depends on it for health checks and LB-level failover routing; pair its probes with the redundancy here to remove SPOFs.
• consistency-coordination — owned-concept lives in it: the consistency consequences of failover (un-replicated writes lost, quorum under partition) are decided there.
• api-design — depends on its idempotency-key contract before any retry of a write is safe.
• caching — pairs with graceful degradation as a fallback source; owned-concept lives in it for the cache-expiry stampede (vs. the recovery herd here).
• system-design — feeds into the orchestrator; this block is its step-5 failure-mode check.

References

• references/deep-dive.md — circuit-breaker state machine, backoff/jitter formulas, retry budgets, the five rate-limiting algorithms (token bucket, leaky bucket, fixed/sliding window) with distributed-counter and race-condition handling, bulkhead sizing, SPOF analysis and failover modes. Read when designing the resilience layer in detail.
• references/providers/{generic,aws,azure,gcp,temporal}.md — service mappings, limits, and pitfalls per environment; temporal.md covers durable retries/timeouts and saga compensation as workflow primitives.