llm-routing-ladder

LLM Routing Ladder

When to invoke

• Designing the model-selection strategy for a new Claude app.
• Monthly cost is higher than expected and "use Opus for everything" is the de-facto policy.
• Latency requirements vary across request types and a single model isn't the right answer.
• Evaluating whether a task-routing layer is worth the added complexity.

The core concept

A routing ladder directs each request to the cheapest model that resolves it correctly. The metric is cost-per-resolved-task, not tokens or per-call cost.

Incoming request
    │
    ▼
[Triage classifier — Haiku]  ←─ cheap, fast, binary classification
    │
    ├─ Simple  ──────────────► [Haiku handler]  → resolved? → done
    │
    ├─ Medium  ──────────────► [Sonnet handler] → resolved? → done
    │                                                    │ no
    └─ Complex ──────────────► [Opus handler]   ◄────────┘

Step 1 — Define complexity tiers

Tier	Characteristics	Default model
Simple	Single-turn, factual lookup, <500 token output, no tool use	Haiku
Medium	Multi-step, moderate reasoning, tool use with ≤3 calls	Sonnet
Complex	Long chain-of-thought, multi-tool orchestration, high-stakes output	Opus

Write explicit definitions for your domain — "simple" for a customer support bot differs from "simple" for a code review agent.

Step 2 — Triage classifier design

The triage step is itself a Haiku call (or a heuristic) that returns a tier label:

TRIAGE_PROMPT = """Classify the following user request into one of three tiers:
- simple: factual lookup, one-step task, no reasoning chain needed
- medium: multi-step, moderate reasoning or tool use
- complex: requires deep reasoning, long output, or high-stakes judgment

Return ONLY the tier label. Request: {request}"""

def classify(request: str) -> str:
    resp = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=10,
        messages=[{"role": "user", "content": TRIAGE_PROMPT.format(request=request)}]
    )
    return resp.content[0].text.strip().lower()

Cache the triage result for identical requests (same hash) to avoid paying the classifier cost twice.

Step 3 — Escalation on uncertainty

Simple and medium handlers should detect uncertainty and escalate:

UNCERTAINTY_SIGNALS = [
    "I'm not sure", "I cannot determine", "this requires",
    "outside my knowledge", "I would need more"
]

def should_escalate(response_text: str) -> bool:
    return any(s in response_text for s in UNCERTAINTY_SIGNALS)

Escalation is one tier at a time — Haiku → Sonnet → Opus. Never skip tiers unless the triage classified complex from the start.

Step 4 — Measure cost-per-resolved-task

# Track per request:
# - model used
# - input_tokens, output_tokens, cache_read_input_tokens
# - resolved: bool (from your eval / user feedback)
# Then compute:
cost_per_resolved = total_cost / resolved_count
# Compare across ladder configs; the goal is lower cost_per_resolved, not lower per-call cost

A ladder that escalates 40 % of calls to Opus may be more expensive than running Sonnet everywhere if Haiku's resolution rate is low. Measure before assuming the ladder helps.

Step 5 — Routing ladder + Batch API

For async / offline workloads (nightly reports, bulk classification, eval runs):

• Run the full batch at Haiku first (50 % discount via Batch API).
• Flag low-confidence responses for a second Sonnet pass.
• Only Opus-escalate the residual requiring expert reasoning.
• Batch mode adds ~15 min latency; never use for interactive user requests.

Cost model table (reference — verify current pricing)

Model	Input cost	Cache-read cost	Batch discount
Haiku	low	10 % of input	50 %
Sonnet	medium	10 % of input	50 %
Opus	high	10 % of input	50 %

[verify-at-build] — pricing changes; retrieve from knowledge/model-selection-and-2026-capability-map.md before quoting.

Pitfalls

• Using the routing ladder as an excuse to avoid prompt optimisation — a well-engineered Sonnet prompt often beats a poorly-engineered Opus call at 1/5 the cost.
• Not pinning the triage classifier's model — a Sonnet triage that costs as much as a Haiku resolution defeats the purpose.
• Treating the ladder as set-and-forget — measure resolution rate per tier monthly; model capability changes can shift which tier owns which request type.
• Adding a ladder to a low-volume app (< 1 000 req/day) — the engineering cost rarely pays back below that volume.