coreweave-gpu-node-forensics

CoreWeave GPU Node Forensics

Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc. "NVIDIA" and "Xid" are trademarks of NVIDIA Corporation; Xid semantics are cited from NVIDIA's public documentation.

Triages a dead or degraded GPU on a CoreWeave node in seconds and returns one grounded move — reschedule, reset-gpu, reboot-node, rma, watch, or app-bug-not-hardware — from an NVIDIA Xid code or a pasted dmesg / nvidia-smi blob. The classification is deterministic: a bundled script does the mapping so the agent never guesses whether a card is dead, degraded, or fine.

Overview

A single bad GPU can kill a 64-GPU, multi-day training run — thousands of dollars and days of wall-clock gone — because one rank stalls the whole collective. The expensive mistakes are triage mistakes: RMAing a healthy card for an app bug, restarting a job onto a GPU whose memory error was uncontained, or manually uncordoning a node the lifecycle controller is trying to replace. This skill kills that ambiguity.

The decision logic is grounded in the NVIDIA Xid error catalog (https://docs.nvidia.com/deploy/xid-errors/) and CoreWeave's node-lifecycle / cordon behavior. The math-of-the-matter — which Xid means what, and how row-remapper state overrides it — lives in scripts/triage.py as a table the LLM does not get to re-litigate. Deep domain knowledge (the full code→action table, the row-remap decision, the cordon rules) loads from references/ on demand.

The headline is the Xid 94-vs-95 split. A contained memory error (94) cost you one job restart on a healthy node; an uncontained one (95) means the GPU could not isolate the fault and everything it touched is suspect. Getting that one bit wrong is the difference between a 30-second reschedule and a run that quietly trained on corrupt gradients. The script decides it; the skill never eyeballs it.

This skill is diagnostic, not destructive: it recommends the cordon / drain / reset / RMA next-step but its tools are scoped read-only (nvidia-smi -q, kubectl get, dmesg) — it never runs a reset, a reboot, or an uncordon itself.

Prerequisites

• The failure evidence. Either an Xid number, or a pasted dmesg / nvidia-smi dump. The skill works from a paste alone — no live cluster access required — which is the common case (an operator pastes what the run's logs showed).
• Optional live access for corroboration: kubectl context on the CoreWeave cluster (read-only is enough), and nvidia-smi on the node. If neither is available the skill still triages from the paste.
• python3 to run the deterministic classifier (scripts/triage.py, stdlib only — no dependencies).

No secrets are handled. All commands are read-only queries.

Instructions

The pipeline is capture → classify → act. The classifier is authoritative for the verdict; references/ supplies the "why" when a case needs depth.

Step 1: Capture the evidence

If the user has not already pasted it, ask for (or read) the fault signal. The two richest sources:

dmesg -T | grep -i xid                       # the Xid line(s) with timestamps
nvidia-smi -q -d ROW_REMAPPER,ECC,PERFORMANCE # remap state, ECC counts, throttle reasons

On a CoreWeave node you can also check who owns any cordon before acting:

kubectl get node NODE -o json | jq '{unschedulable: .spec.unschedulable, taints: .spec.taints}'

Step 2: Run the deterministic triage

Feed the Xid code, or the whole blob, to the classifier. Do not classify by hand — the script owns the Xid→action mapping and the row-remap override.

# From an Xid code:
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 95

# With row-remapper state (Xid 63/64 or a DBE):
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 63 --pending yes
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --xid 48 --remap-failure yes

# From a pasted dmesg / nvidia-smi blob (file or stdin):
python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py" --blob /path/to/dmesg.txt
dmesg -T | grep -i xid | python3 "${CLAUDE_SKILL_DIR}/scripts/triage.py"

The script emits a VERDICT block plus a JSON object ({classification, severity, action, why, next_command, cordon_rule?, unverified?}). Add --json for machine-readable output only.

When a blob carries several Xids, the most severe one governs the move and the rest are reported as co_occurring_xids — a co-occurring app-side Xid 43 next to a hardware Xid 79 does not soften the "reboot the node" verdict.

Step 3: Read the verdict and act on the action

Present the action and the next_command to the user in plain language. The six actions and what each means live in ${CLAUDE_SKILL_DIR}/references/xid-triage-table.md. Load it when the user wants the full table or asks about an Xid not in the summary.

• reschedule — restart the failed rank; the node stays in service.
• reset-gpu — drain the GPU and reset it; re-run any job that shared it.
• reboot-node — the card is off the bus; only a bare-metal reboot returns it.
• rma — terminal hardware fault; the card must be replaced.
• watch — correctable / trending; monitor, do not act yet.
• app-bug-not-hardware — the app faulted, not the GPU. Do NOT RMA.

Step 4: The 94-vs-95 headline (load-bearing — get this right)

If the Xid is 94 or 95, state the containment explicitly, because the two look almost identical in the logs and lead to opposite actions:

• Xid 94 (CONTAINED) → reschedule. The error was isolated to the app's context; the GPU and node are healthy. Restart the job. Do not reset or RMA.
• Xid 95 (UNCONTAINED) → reset-gpu. The GPU could not isolate it; every context it touched is suspect. Drain, reset, and re-run anything that shared the card — otherwise the run may continue on corrupt state.

Step 5: Row-remap (Xid 63 / 64 / 48) — routine vs terminal

For any ECC/DBE or row-remap Xid, the nvidia-smi -q -d ROW_REMAPPER fields override the base verdict. Pass them in (--pending, --remap-failure) and let the script decide. The rule:

• Remapping Failure Occurred: Yes → rma (terminal — sparing failed).
• Pending: Yes (no failure) → reset-gpu (routine — applies on reset).

Full logic + how to read the histogram: ${CLAUDE_SKILL_DIR}/references/row-remap-decision.md.

Step 6: The cordon hard rule (never violate)

Whenever the action is a hardware move (reset-gpu, reboot-node, rma), the verdict carries a cordon_rule. Surface it verbatim:

Never manually uncordon a CoreWeave health cordon — the node-lifecycle controller owns cordon/uncordon and is driving remediation. Uncordoning re-admits the run onto known-bad hardware and races the controller.

Determine cordon provenance before touching schedulability. Full guidance: ${CLAUDE_SKILL_DIR}/references/cordon-rules.md.

Output

• A verdict — the deterministic {classification, severity, action, why, next_command} for the governing signal, presented in plain language with the single next command an operator runs.
• The 94-vs-95 call stated explicitly when either fires — contained → reschedule, uncontained → reset — never conflated.
• The cordon rule attached to every hardware action, verbatim, so nobody uncordons a node the controller is replacing.
• Co-occurring Xids listed when a blob carried several, with the most-severe one named as governing.
• [unverified] hedges surfaced honestly — e.g. CoreWeave's exact auto-RMA thresholds and health-cordon taint strings are not publicly published, and the skill says so rather than inventing a number.

Error Handling

Situation	Cause	Response
Unmapped Xid	Xid not in the triage table	Script returns a conservative reset-gpu default, flags it [unverified], and points at the NVIDIA catalog. Do not RMA on an unmapped Xid.
Xid 94 vs 95 ambiguity	Both are "ECC error" in casual logs	Never infer from prose — use the exact code. 94 = contained (reschedule), 95 = uncontained (reset).
Row-remap without an Xid	Only nvidia-smi pasted	Script reads Remapping Failure Occurred / Pending and decides from those alone.
App-side Xid mistaken for hardware	Xid 13/31/43/45	Classed app-bug-not-hardware; the fix is compute-sanitizer on the job, never an RMA.
Multiple Xids in one blob	Cascade (e.g. 43 then 79)	Most-severe governs; others listed as co_occurring_xids.
Empty / no signal	Blob has no Xid, remap, or thermal line	Script returns watch and asks for fresh dmesg + nvidia-smi -q evidence.
Thermal throttle, not a fault	HW/SW Thermal Slowdown : Active	Classed watch — healthy but hot; investigate cooling, not the card.
Tempted to uncordon a stalled node	Health cordon owned by the controller	Refuse. Surface the cordon rule; let remediation run.

Examples

Example 1: "Xid 94 on one rank — do I need to replace the GPU?"

VERDICT: Xid 94 — Contained ECC/memory error [severity: HIGH]
ACTION:  reschedule
WHY:     The error was CONTAINED to the faulting application's context — the GPU
         and node are healthy. Just reschedule/restart the job; no reset or RMA needed.
NEXT:    Restart or reschedule the failed rank; the node stays in service.

No replacement. The containment did its job — restart the rank and keep the node.

Example 2: "Xid 95 — same run, different node"

VERDICT: Xid 95 — Uncontained ECC/memory error [severity: CRITICAL]
ACTION:  reset-gpu
WHY:     The error was UNCONTAINED — the GPU could not isolate it, so every context
         it touched is suspect. Drain the GPU and reset it; treat all in-flight work as corrupt.
NEXT:    Cordon + drain, GPU-reset (nvidia-smi -r) or node reset; re-run any job that shared this GPU.
CORDON:  Do NOT manually uncordon a CoreWeave health cordon — the node-lifecycle
         controller owns cordon/uncordon. Let it drain and replace the node.

Opposite of Example 1 despite looking identical in the logs: drain, reset, and re-run the shared work — do not just restart.

Example 3: "GPU fell off the bus"

python3 scripts/triage.py --xid 79 → reboot-node. The card is unreachable on the PCIe bus and returns only after a bare-metal reboot; expect the controller to cordon and reboot — let it, and RMA only if it recurs after the reboot.

Example 4: "Xid 63 with a remap failure"

python3 scripts/triage.py --xid 63 --remap-failure yes → rma. The remapper tried to swap in a spare row and physically could not — terminal. Cordon, drain, open the RMA. (The script flags that CoreWeave's exact auto-RMA threshold is [unverified].)

Example 5: "Xid 43 killed my job"

python3 scripts/triage.py --xid 43 → app-bug-not-hardware. Software-induced fault — the fix is in the application (run it under compute-sanitizer), not an RMA. Pulling the card would waste healthy hardware and never fix the job.

Resources

• ${CLAUDE_SKILL_DIR}/references/xid-triage-table.md — the full Xid→action table, the six actions, the 94-vs-95 distinction, cited to the NVIDIA Xid catalog.
• ${CLAUDE_SKILL_DIR}/references/row-remap-decision.md — the ROW_REMAPPER routine-vs-terminal decision and how to read the histogram.
• ${CLAUDE_SKILL_DIR}/references/cordon-rules.md — CoreWeave health-cordon ownership and the safe operator path per action.
• NVIDIA Xid Errors reference
• NVIDIA GPU Memory Error Management (row remapping)