add-llm-fallbacks

Add LLM fallbacks and resilience

LLM providers rate-limit, time out, 500, and occasionally go down. A single unguarded call means your app goes down with them. Add layered resilience, and instrument it so you can see when it kicks in.

The layers (add all of them)

1. Timeouts. Every call needs a sane timeout. An unbounded call can hang a request forever. Set it based on your p95 latency plus headroom.
2. Retries with backoff. Retry transient failures (429, 500, 503, timeouts) with exponential backoff + jitter. Cap the number of retries and cap the total wait, and honor Retry-After when present. A naive retry loop turns a rate limit into a retry storm (and a big bill, see reduce-llm-cost).
3. Do not retry non-transient errors. 400s (bad request), auth errors, and content-policy refusals will fail again. Retrying them wastes time and money.
4. Model / provider fallback. If the primary model errors or is down, fall back to a secondary (same provider different model, or a different provider). A gateway (LiteLLM, Portkey) gives you this with config instead of code.
5. Graceful degradation. When everything fails, return a sensible fallback (cached answer, a "try again shortly" message, a simpler non-LLM path) instead of a stack trace.

Instrument it (so resilience is not invisible)

The failure mode of good resilience is that it hides problems. Emit telemetry so you still see them:

• A span/event when a retry happens (and how many).
• A span/event when a fallback fires (primary -> secondary).
• Error rate per provider/model and fallback rate on a dashboard, with alerts (see set-up-drift-alerts).

If your fallback rate silently climbs to 30%, you want to know, not find out from the bill or a user.

Watch the cost/quality of fallbacks

A fallback model may be cheaper-but-worse or pricier. Track cost and (ideally) quality on the fallback path too, so a provider outage does not silently degrade your app for hours.

Verify

• Kill the primary provider (bad key / block the host) and confirm the app fails over, not over.
• Simulate a 429 and confirm backoff + Retry-After are honored, with a capped total wait.
• Confirm retries and fallbacks show up in traces/metrics.

Anti-patterns

• No timeout (one hung call ties up a worker).
• Retrying 4xx/auth/policy errors (they will not succeed; you just wait and pay).
• Fixed-delay retries with no cap (retry storm on a rate limit).
• Silent fallbacks with no telemetry (you are degraded and blind to it).
• A fallback model nobody evaluated (you fail over into worse answers).

Grounding

Standard distributed-systems resilience patterns applied to LLM calls: exponential backoff with jitter (AWS Architecture guidance), retry/timeout/circuit-breaker (Nygard, Release It!). Gateways implement provider fallback: LiteLLM, Portkey.