check-routing-health

Check Routing Health

Use this worker when the developer wants to know whether Understudy routing is causing errors, which workloads are routed, whether a declared route is really taking effect, or where their gateway usage and spend goes. These are read-only hosted endpoints that answer "is this us?" without asking the team.

Checked against existing skills: use-understudy-gateway owns auth, routing setup, and route writes; ramp-and-verify owns production traffic changes. This skill owns read-only diagnostic queries against the reporting surface and does not write routes or change traffic.

Safety Gates

These endpoints are read-only and carry no side effects. They do not change routes, traffic percentages, or provider configuration. Do not print the full sk_* key in output — mask it to the last 4 characters. Use understudy run to inject credentials into child processes instead of pasting keys into shell commands.

Vocabulary — use these words, exactly

Everything you say to the user must use the canonical routing vocabulary:

• Route outcomes (route_shares): primary — the customer-requested model was served (includes catalog-by-name); understudy — an Understudy-configured route moved the traffic; fallback — a recovery re-issue after a routed attempt failed.
• Declared config (declared): pin | steer | none, plus the split percent (split_pct).
• Provider labels: anthropic | openai | managed — nothing else.
• A served model of understudy-managed is a managed model whose catalog mapping hasn't synced yet — a fail-closed placeholder, not an error.
• Never use "passthrough", "BYO", or "relay" in output to users. Older responses (the legacy routing-status endpoint) still emit some of these words — translate, don't echo.

Prerequisites

The developer must be signed in (understudy status --json shows signed_in: true) and have a project configured. If not, route to ../use-understudy-gateway/SKILL.md for auth setup first.

Resolve CLI

Prefer the installed understudy binary. If it is unavailable inside a repo checkout, run through the package script:

npm run build
node dist/bin.js status --json

Flow

1. Confirm auth and project:

   understudy status --json
   

   Extract org_id and project_slug from the output. Resolve the project slug to a project id via understudy projects list --json if needed — the API path requires the proj_... id, not the slug. If not signed in, route to ../use-understudy-gateway/SKILL.md.

2. Ground in volume first — one org-wide call. Before analyzing or recommending anything, rank workloads by spend and request count across the whole org. understudy run spawns the child without a shell, so wrap the command in sh -c '...' — the injected $UNDERSTUDY_API_KEY / $UNDERSTUDY_ORG_ID only expand inside that child shell (bare understudy run -- curl would send the literal string $UNDERSTUDY_API_KEY as the bearer):

   understudy run -- sh -c 'curl -s \
     -H "Authorization: Bearer $UNDERSTUDY_API_KEY" \
     "https://api.understudylabs.com/admin/v1/orgs/$UNDERSTUDY_ORG_ID/reporting?window=7d&group_by=workload"'
   

   Rank by customer_cost_usd and requests (group_by defaults to project — use that first when the org has many projects, then workload). For a tight incident window add granularity=minute (ranges up to 24h); drill down with the project_id / workload_id filters. Every statement you make must be grounded in that volume ranking — lead with the workloads that carry the spend and traffic. Do not anchor on low-leverage generic advice about workloads that barely run.

3. Pull the unified per-workload view (same sh -c wrapper):

   understudy run -- sh -c 'curl -s \
     -H "Authorization: Bearer $UNDERSTUDY_API_KEY" \
     "https://api.understudylabs.com/admin/v1/orgs/$UNDERSTUDY_ORG_ID/projects/<project-id>/workload-status?window=24h"'
   

   One row per workload: status (healthy | degraded | idle), the declared config, observed route_shares, rerouted_pct, served_models, error_rate, and example_request_ids. Field details in reference.md.

4. Interpret, in priority order of the volume ranking from step 2:

   • status: degraded — the 5xx rate crossed the threshold. Report the workload-level error_rate and the example_request_ids. Do NOT name a failing upstream from served_models[] — it carries request shares only, no per-provider error counts, so traffic labels cannot say which provider produced the 5xxs. Leave provider attribution to the Understudy team and quote the request ids.
   • Declared-vs-observed drift is a finding — normalize units first. declared.split_pct is 0–100; route_shares.understudy is a 0..1 share. Compare split_pct / 100 against the share — never the raw numbers (declared 30 vs observed 0.28 is healthy, not drift). Flag drift when split_pct / 100 − route_shares.understudy exceeds ~0.1; the loudest case is a declared split with an observed share of ~0 — tell the user plainly their routing config is declared but not actually taking effect.
   • rerouted_pct (= route_shares.understudy) is THE number to watch during a ramp or cutover — compare it against declared.split_pct / 100.
   • Non-zero route_shares.fallback means routed attempts are failing and being recovered — upstream instability on the routed arm.
   • Error rate above ~2% is worth investigating; a workload with declared.routed: none and errors is provider-side, not routing-caused.

5. If example_request_ids are present, offer to look them up:

   understudy captures get <request-id> --project <project>
   

   If they are empty (or the team needs more ids), pull recent capture metadata for the project — request ids, timestamps, sizes; metadata only, capture content stays behind the dashboard login (reference.md § Captures metadata list):

   understudy run -- sh -c 'curl -s \
     -H "Authorization: Bearer $UNDERSTUDY_API_KEY" \
     "https://api.understudylabs.com/admin/v1/orgs/$UNDERSTUDY_ORG_ID/projects/<project-id>/captures?limit=25"'
   

6. If any call fails or a number looks wrong, capture the x-understudy-request-id response header and quote it when reporting the problem to the Understudy team — it is the join key on their side.

7. If issues are found and the developer wants to roll back, route to ../ramp-and-verify/SKILL.md or clear the route immediately with understudy routes clear <workload> --project <project>.

Project-scoped alternatives. usage-summary remains the detail tool within one project when the org-wide view isn't enough: it carries what org reporting doesn't — the computed cache_read_pct share, a per-group error_rate, and multi-dimension group_by (e.g. workload,day). Raw cache token counts are in both. See reference.md § Usage summary. The older routing-status, provider-health, and status endpoints still exist but are deprecated — see the legacy section of reference.md. Only fall back to them if workload-status or usage-summary return 404 (an older deployment).

Output Standard

End with:

• project/org context (without revealing the full key);
• the volume ranking (top workloads by spend/requests) that grounds the answer;
• per-workload status and any declared-vs-observed drift findings, in canonical vocabulary;
• the window queried and when the data was generated;
• recommended next action (investigate a request ID, adjust the window, roll back a route, or confirm healthy) — with any request ids quoted.

References

• reference.md — endpoint details, response shapes, field descriptions, and the deprecated legacy endpoints.
• ../use-understudy-gateway/SKILL.md — auth setup, route writes, and gateway inference.
• ../ramp-and-verify/SKILL.md — production ramp/rollback when diagnostics reveal a problem.