coreweave-fabric-diagnostics

CoreWeave Fabric Diagnostics

Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Detects when a CoreWeave multi-node GPU job has silently fallen off the InfiniBand fabric onto TCP — the failure that makes distributed training run at a fraction of the hardware's speed while every GPU keeps billing at the full rate — and gives the exact fix.

Overview

On a CoreWeave multi-node job, NCCL should carry collectives over InfiniBand with GPUDirect RDMA (NET/IB). If any one of three conditions is missing, NCCL silently falls back to TCP sockets (NET/Socket): the job still runs, still converges, and raises no error — but all-reduce throughput collapses (commonly cited as 5-20x slower [nccl]) because it now crosses the Ethernet control plane instead of the 400 Gb/s-class fabric. You keep paying full GPU rate for a multi-node run that performs like a badly-connected one. This is the single highest-dollar invisible failure on the platform, and nothing in the default output flags it.

The diagnosis is deterministic: the bundled scripts/fabric-check.py greps the pasted NCCL_DEBUG=INFO log for the decisive Using network line (and NET/IB vs NET/Socket), parses the pod-spec's resources block for the RDMA device request, reads ibstat port state, and echoes any all_reduce_perf bus bandwidth — then emits a VERDICT with the rdma_engaged/transport call, the missing conditions, and the fix. The LLM never eyeballs which transport is in use; the script decides. Deep grounding lives in references/, loaded only when a leg of the diagnosis needs it.

Prerequisites

• An NCCL_DEBUG=INFO log from the actual run — the primary signal. Re-run the job (or one rank) with NCCL_DEBUG=INFO set and capture stderr. The decisive line is Using network IB (good) vs Using network Socket (the fallback). This is the one input the skill really needs; everything else corroborates.
• Optional, for a full diagnosis: the pod/job spec (kubectl get pod NAME -o yaml) to check the RDMA device request; ibstat output from the node for port health; and all_reduce_perf results from CoreWeave's nccl-tests to measure bus bandwidth.
• python3 to run the deterministic checker (stdlib only).
• kubectl (read-only) if corroborating the live pod spec / node cordon state.

Authentication. Nothing secret is read. If the pod spec is pulled live, kubectl uses the existing $KUBECONFIG; the skill only ever runs kubectl get (read-only) — it never cordons, drains, or applies.

The three required conditions (all must hold, or NCCL falls back)

1. The RDMA device is requested in BOTH resources.requests AND resources.limits (rdma/ib: 1). If it is in only one — or absent — the device plugin does not inject the IB device into the pod and NCCL never sees a HCA. [unverified — the exact resource key (e.g. rdma/ib) depends on the installed RDMA device-plugin config; confirm with kubectl describe node / kubectl get node -o yaml.]
2. NCCL_IB_HCA=ibp and NCCL_SOCKET_IFNAME=eth0 are set (CoreWeave's documented values [cw]) — unless you launch via the MPI Operator, which manages this network config for you [nt].
3. NCCL_DEBUG=INFO then confirms NET/IB (ideally a GPU Direct RDMA Enabled line). If it shows NET/Socket / Using network Socket, RDMA is not engaged.

Full checklist with verification commands: references/rdma-engagement-checklist.md.

Instructions

The pipeline is gather → verdict → fix → confirm. The script does the transport call; references/ carry the grounding:

1. Gather the NCCL_DEBUG=INFO log (required) plus any pod-spec / ibstat / all_reduce_perf output you have. Concatenate them into one paste — the checker keys on each signal independently.
2. Run the deterministic checker to get the VERDICT.
3. If the verdict is fallback (Socket), apply the three-condition fix and re-run.
4. If the verdict is IB but degraded, chase the degraded signal (port down / low busbw).
5. On NVSwitch systems with a stuck fabric, use the Fabric Manager reset order.

Step 1: Gather the evidence

The log is the load-bearing input. If the user has not run with NCCL_DEBUG=INFO, tell them to — without it, transport selection is unknowable. To pull the live pod spec:

kubectl get pod "$POD" -o yaml > pod.yaml

Step 2: Run the deterministic checker (it makes the call, not the model)

Pipe everything you gathered to fabric-check.py. It greps for the decisive Using network line, the resources block, ibstat state, and any Avg bus bandwidth:

cat nccl-debug.log pod.yaml ibstat.txt allreduce.txt 2>/dev/null | \
  python3 scripts/fabric-check.py

The verdict names rdma_engaged (yes/no/partial/unknown), the transport in use, the missing conditions, and the fix. Use --json to capture the structured result for further processing. Reading the log by eye is what this step exists to prevent — see references/nccl-debug-reading.md for what each line means.

Use Glob to gather multiple pasted log files when a run spans several ranks, Write the verdict report to the working directory, and Edit it to refine the fix as the user iterates on the manifest.

Step 3: If the verdict is fallback (NET/Socket) — apply the three-condition fix

This is the money case. Fix in order (the checker prints the same list):

1. Add rdma/ib: 1 to both resources.requests and resources.limits.
2. Set NCCL_IB_HCA=ibp and NCCL_SOCKET_IFNAME=eth0 (or launch via the MPI Operator).
3. Re-run with NCCL_DEBUG=INFO and confirm the log now shows NET/IB + GPU Direct RDMA Enabled, not NET/Socket.

If the log shows NCCL_IB_DISABLE=1, that alone forces sockets — set it to 0 (RoCE and IB both need the IB verbs transport enabled [env]).

Step 4: If the verdict is IB-but-degraded — chase the degraded signal

RDMA can be engaged yet slow. Two corroborating checks:

• ibstat — every port must read State: Active / Physical state: LinkUp. A port Down/Polling, or a link that flaps, drags the whole collective; CoreWeave auto-cordons flapping links, so a shrinking node count mid-run is a fabric symptom.
• all_reduce_perf bus bandwidth — compare the reported busbw against CoreWeave's published nccl-tests manifest baseline for your GPU count + NCCL version [nt]. Do not compare against a fixed number: the baseline moves with GPU type, node count, NCCL version, and SHARP. The checker echoes the observed figure tagged [unverified vs baseline] precisely so nobody reads it as a hard pass/fail.

Details + the busbw-vs-algbw distinction: references/allreduce-baseline.md.

Step 5: NVSwitch systems — Fabric Manager reset order

On NVSwitch/NVLink systems, a wedged fabric shows up as NVLink/NVSwitch errors rather than IB fallback. The safe reset order is stop Fabric Manager → reset the GPUs → start Fabric Manager, never the reverse:

sudo systemctl stop nvidia-fabricmanager
sudo nvidia-smi -r            # GPU reset
sudo systemctl start nvidia-fabricmanager

[unverified — service unit name and reset support vary by image/driver; on managed CoreWeave nodes prefer opening a support ticket / cordoning over an in-place reset.]

Output

• A VERDICT line stating whether RDMA is engaged, the transport actually in use, and — for the fallback case — the plain-language cost framing (running on TCP, paying full GPU rate for a fraction of the throughput).
• The missing-conditions list — which of the three required conditions is absent, each one sufficient on its own to force the fallback.
• The ordered fix — the rdma/ib-in-requests-AND-limits change, the env vars, and the re-verify step.
• Degraded-fabric signals when RDMA is engaged but slow — down/flapping IB ports and the observed busbw (tagged [unverified vs baseline]).

Error Handling

Error	Cause	Solution
Verdict is unknown	No NET/IB / NET/Socket / Using network line in the paste	Re-run the job with NCCL_DEBUG=INFO and capture stderr; without it transport is unknowable.
Verdict Socket but the pod "has RDMA"	rdma/ib in limits only (or only requests)	Add it to BOTH blocks; the device plugin injects the IB device only when the resource is requested.
NET/IB present yet training still slow	GDR not actually enabled; nvidia-peermem unloaded → traffic stages through host memory	Confirm a GPU Direct RDMA Enabled line; verify nvidia-peermem is loaded on the node [nccl].
busbw "looks low"	Compared against a wrong/guessed baseline	Compare only against CoreWeave's nccl-tests manifest baseline for your GPU count + NCCL version; the number is workload/version-dependent.
Nodes drop out mid-run	Flapping IB link → CoreWeave auto-cordon	Check ibstat for Physical state != LinkUp; the cordoned node's link is the cause, not your job.
rdma/ib resource not schedulable	Wrong resource key for the installed device plugin	Confirm the exact key with kubectl describe node (search the Allocatable list) and substitute it.

Examples

Example 1: "Our 4-node H100 training run got slow — is RDMA even working?"

The user pastes an NCCL_DEBUG=INFO excerpt plus the pod spec. The checker finds Using network Socket and rdma/ib only in requests, and verdicts:

### VERDICT: RDMA is NOT engaged -- NCCL fell back to TCP (NET/Socket). Multi-node collectives are running over the Ethernet control plane, commonly 5-20x slower for the same GPU-hours -- you pay full GPU rate for a fraction of the throughput, and NCCL raised no error.

- RDMA engaged: **no**
- Transport in use: **Socket**
- Missing conditions (each one alone forces a silent TCP fallback):
    - `rdma/ib` missing from resources.limits
    - `NCCL_IB_HCA` not set (e.g. `ibp`) -- unless the MPI Operator manages it

**The fix (in order):**
1. Request the RDMA device in BOTH requests AND limits: `rdma/ib: 1` (if it is in only one, the device plugin will not inject the IB device).
2. Set `NCCL_IB_HCA=ibp` and `NCCL_SOCKET_IFNAME=eth0` (CoreWeave values), or let the MPI Operator manage them.
3. Re-run with `NCCL_DEBUG=INFO` and confirm the log now shows `NET/IB` and `GPU Direct RDMA Enabled` -- not `NET/Socket` / `Using network Socket`.
4. Confirm each IB port is `State: Active` / `Physical state: LinkUp` via `ibstat`; a flapping link gets auto-cordoned by CoreWeave.

Example 2: "RDMA is on but all-reduce bandwidth seems low"

The log shows NET/IB and GPU Direct RDMA Enabled, so the checker returns rdma_engaged: yes. It then surfaces the ibstat port that reads Physical state: Polling as a degraded signal and echoes the observed busbw tagged [unverified vs baseline], directing the user to compare against CoreWeave's nccl-tests manifest for their GPU count + NCCL version rather than a guessed number.

Resources

• references/rdma-engagement-checklist.md — the three required conditions + how to verify each, cited.
• references/nccl-debug-reading.md — reading NCCL_DEBUG=INFO: NET/IB vs NET/Socket, the decisive Using network line, GDR.
• references/allreduce-baseline.md — all_reduce_perf busbw/algbw and why the baseline is never hardcoded.
• Sibling: coreweave-gpu-cost-leak-hunter dollarizes idle/right-sizing spend; this skill finds the throughput leak (fabric fallback) that a cost report cannot see.