serving-llms-on-instinct

Serving LLMs on AMD Instinct

Get a vLLM endpoint running on AMD Instinct GPU hardware.

Prerequisites

• ROCm driver and amd-smi installed on the GPU host
• Docker running and accessible (check with docker ps)
• /dev/kfd and /dev/dri present on the GPU host
• HuggingFace token in HF_TOKEN env var (required for gated models; not required for Qwen3 or Gemma). For gated models (Llama 3.2, Gemma, etc.), the HF token must belong to an account that has accepted the model's license at huggingface.co/<model_id>. A valid token without license acceptance will fail with an opaque "Engine core initialization failed" error.
• For remote GPU: SSH key access configured (ssh <user>@<host> must work without a password prompt). If only password access is available, set up keys first: ssh-copy-id <user>@<host>

Data files

Read these files directly to get model and GPU configuration:

• data/recipes_cache.json -- model configs synced from vllm-project/recipes. Each entry under models.<HF_ID>.recipe contains the full recipe with model.base_args, model.base_env, features.tool_calling.args, features.reasoning.args, hardware_overrides.amd.extra_args, hardware_overrides.amd.extra_env. The top-level docker_image field has the latest resolved vLLM ROCm image.

• data/gpu_overrides.json -- GPU-specific configuration. Contains docker_flags (mandatory for all AMD Instinct), gpu_configs keyed by gfx_version with env_defaults and workarounds, and legacy_models for models not yet in vLLM recipes.

• data/blacklist.json -- models in vLLM recipes that cannot be served as LLM endpoints. Includes diffusion/image/audio generation models, embedding models, rerankers, ASR models needing audio pipelines, and models requiring unreleased vLLM nightly builds. Check this before attempting to serve a model. If the user requests a blacklisted model, explain why it won't work and suggest an alternative.

If the user doesn't specify a model, default to Qwen/Qwen3.5-9B: dense multimodal with MTP, Apache 2.0 license (no HF token needed), fits on a single GPU, strong reasoning and tool-calling.

Step 1: Detect the GPU

python3 scripts/detect.py
# Remote:
python3 scripts/detect.py --host user@hostname

Returns JSON with gfx_version, vram_gb, gpu_count, rocm_version.

gfx_version	Hardware	VRAM
gfx950	MI350X / MI355X	288 GB HBM3E
gfx942	MI300X (192 GB) / MI325X (256 GB) / MI300A (128 GB)	varies

If gfx_version is unknown: amd-smi ran but found no GPU. Check lsmod | grep amdgpu.

Step 2: Validate the environment

python3 scripts/validate.py --auto-fix
# Remote:
python3 scripts/validate.py --auto-fix --host user@hostname

Returns JSON with ready (bool), errors, warnings, fixes_applied. Do not proceed if ready is false.

Step 3: Refresh recipes (if stale)

Check fetched_at in data/recipes_cache.json. If older than 24 hours or the file is missing, refresh:

python3 scripts/sync_recipes.py

This shallow-clones vllm-project/recipes from GitHub and fetches the latest Docker tag from Docker Hub. Takes ~10 seconds. If it fails, the existing cache still works.

Step 4: Construct the Docker command

Read data/recipes_cache.json and data/gpu_overrides.json directly. Build the Docker command by combining:

1. Docker flags from gpu_overrides.json > docker_flags (mandatory for all AMD GPUs)
2. HF cache mount: -v ~/.cache/huggingface:/root/.cache/huggingface (if a shared model cache directory exists on the host, check whether models--* directories are at the cache root or inside a hub/ subdirectory -- mount accordingly to /root/.cache/huggingface or /root/.cache/huggingface/hub)
3. Port: -p <port>:<port> (default 8000)
4. Environment variables: merge gpu_configs.<gfx_version>.env_defaults with the recipe's model.base_env and hardware_overrides.amd.extra_env. Always add --env HF_TOKEN=${HF_TOKEN}.
5. Docker image: use docker_image from recipes_cache.json top level (unless the model needs a pinned image, e.g. GLM-4.5 needs v0.15.1). If the user specifies a Docker image version, check it against the recipe's model.min_vllm_version. Warn if the image is older -- the model may crash on startup with an opaque "Engine core initialization failed" error.
6. Model ID: --model <HF_ID>
7. vLLM args: combine the recipe's model.base_args + hardware_overrides.amd.extra_args + features.tool_calling.args + features.reasoning.args. Add --enable-auto-tool-choice if not present. For multi-GPU, add --tensor-parallel-size N (see VRAM estimation below). For MoE models on multi-GPU, also add --distributed-executor-backend mp.
8. Port arg: --port <port>

If the exact model ID is not in recipes_cache.json, check for a base model match by stripping date/version suffixes (e.g., Kimi-K2-Instruct matches Kimi-K2-Instruct-0905). Use the base model's recipe if found.

If no recipe match, check legacy_models in gpu_overrides.json. If not there either, use a generic config with --enable-auto-tool-choice --trust-remote-code --tool-call-parser hermes.

Precision variant selection: Recipes may offer variants (default, fp8, nvfp4). Check gpu_configs.<gfx_version>.precision.native in gpu_overrides.json before selecting a variant. On gfx942 (MI300X), only bf16, fp16, fp8_fnuz, and int8 are hardware-native. MXFP4 and NVFP4 compute is emulated (dequant to BF16 during matmul), but weights stay compressed in VRAM so quantized models still fit in less memory. On gfx950 (MI350X), MXFP4 is hardware-native.

VRAM estimation and fit check: Before constructing the Docker command, estimate whether the model fits the available hardware:

python3 scripts/estimate_vram.py --model-id <HF_ID> --vram-gb <per_gpu_vram> --tp <N>

This queries the HuggingFace Hub API (no model download) and returns JSON with:

• weight_memory_gb -- total weight size
• kv_cache_bytes_per_token -- KV cache cost per token at BF16
• fit.weights_fit -- whether weights fit at the given TP
• fit.recommended_max_model_len -- max context the GPU can serve
• fit.context_limited -- true if KV cache limits context below the model's native max
• fit.min_tp_required -- minimum TP needed (only if weights don't fit)

Understanding the overhead: The script reserves ~4 GB for vLLM's runtime overhead (activation profiling, HIP graph capture, internal buffers). During startup, vLLM runs a profiling forward pass to measure peak activations, then captures HIP graphs for optimized decode. This startup peak is higher than steady-state. The remaining_for_kv_gb field reflects what's left after weights and this overhead.

Use remaining_for_kv_gb to decide:

1. remaining_for_kv_gb >= 6: safe to run. If context_limited: true, add --max-model-len <recommended_max_model_len> to the vLLM args. Mention the FP8 KV cache option (--kv-cache-dtype fp8) if the user needs longer context (fit.max_seq_len_fp8_kv shows the gain).
2. remaining_for_kv_gb between 2 and 6: tight but worth trying. Launch normally. If vLLM OOMs during HIP graph capture (check container logs for "out of memory" after "capturing CUDA/HIP graphs"), retry with --enforce-eager added to the vLLM args. This skips graph capture and frees 1-2 GB. The only cost is slightly higher decode latency.
3. remaining_for_kv_gb < 2: too tight. Will likely OOM during the activation profiling step. Do not attempt.
4. weights_fit: false with multiple GPUs: re-run with --tp <min_tp_required> and check again.
5. weights_fit: false, not enough GPUs: look for quantized alternatives in this order: a. Recipe variants: the recipe may have fp8 or mxfp4 variants with a different model_id that points to a quantized checkpoint. b. Same provider: many providers release quantized versions alongside the base model (e.g. Qwen/Qwen3.5-122B-FP8 from Qwen). Search HuggingFace for <provider>/<model-name> with FP8/GPTQ/AWQ suffixes. c. AMD quantized: AMD's Quark team publishes quantized models under the amd/ org on HuggingFace (e.g. amd/Kimi-K2-Instruct-w-mxfp4-a-fp8). Search for amd/<model-name> variants. Run estimate_vram.py on the quantized model ID to verify it fits, then use that model ID instead.
6. Still doesn't fit: tell the user the model requires more VRAM than available and suggest either a smaller model or multi-GPU hardware. Do not attempt to launch.

Docker command template:

docker run -d --name vllm-<model-slug> \
  <docker_flags> \
  -v <hf_cache_mount> \
  -p <port>:<port> \
  --env <key>=<value> (for each env var) \
  --env HF_TOKEN=${HF_TOKEN} \
  <docker_image> \
  --model <model_id> \
  <vllm_args> \
  --port <port>

Step 5: Confirm with the user

Before launching, present a summary and ask the user to confirm:

• Model: full HuggingFace ID (e.g. Qwen/Qwen3.5-122B-Instruct)
• Precision: variant being used (e.g. BF16, FP8) and why
• Weight memory: from estimate_vram.py
• GPU: detected hardware and VRAM
• TP: tensor parallelism degree (1, 2, 4, 8)
• Context: max achievable context length (and whether it's limited)
• Port: which port the endpoint will be on

If a quantized alternative was selected (Step 4 fit check), explain that the original model doesn't fit and which alternative is being used.

Wait for the user's confirmation before proceeding.

Step 6: Launch and verify

Before launching, check for port conflicts:

ss -tlnp 2>/dev/null | grep ':<port> '

If a Docker container is on that port, stop it with docker rm -f <name>.

Run the Docker command. Then poll health using this loop:

while docker inspect --format='{{.State.Running}}' <container_name> 2>/dev/null | grep -q true; do
  curl -sf http://localhost:<port>/health && echo "READY" && exit 0
  sleep 60
done
echo "FAILED -- container exited"

A 503 during loading is normal. Choose the polling strategy based on model size (weight memory from hf-mem):

• Small models (< 100 GB weights): run the poll as a blocking command with the Bash tool's timeout set to 600000 (10 minutes). Most cached models are ready within 2-5 minutes.
• Large models (>= 100 GB weights): run the poll with the Bash tool's run_in_background set to true. Then use TaskOutput with block: true and timeout: 600000 to wait up to 10 minutes per check. If the task is still running after that, call TaskOutput again with the same parameters. This uses only 1 turn per 10-minute wait instead of burning a turn every check. The background loop runs until the container is healthy or dies.

After health returns 200, send a warmup request (triggers HIP kernel compilation, ~40-45 seconds on gfx942):

curl -s http://localhost:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<model_id>","messages":[{"role":"user","content":"say hi"}],"max_tokens":5}'

After the warmup succeeds, present a connection table so the user can call the endpoint immediately:

Field	Value
Model	<model_id>
Served model name	<served-model-name or model_id>
Base URL	http://<host>:<port>/v1
API key	none (local)
Port	<port>
Tensor parallel	<tp>
Max context	<context>
GPU	<detected GPU>

Then give a ready-to-run example using those exact values:

curl -s http://<host>:<port>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<model_id>","messages":[{"role":"user","content":"Hello"}]}'

Remote vs. local

All scripts accept --host user@hostname. When given, they SSH to the target. Set ROCM_SSH_HOST and ROCM_SSH_USER env vars to avoid passing --host every time.

For remote Docker commands, run them over SSH:

ssh user@host 'docker run -d ...'

Use localhost for health/warmup curl URLs (curl runs on the remote host).

Gotchas

CUDA_VISIBLE_DEVICES set to empty string -- ROCm maps this variable to HIP_VISIBLE_DEVICES. Setting it to an empty string hides all GPUs. CUDA_VISIBLE_DEVICES=0,1 works fine for restricting GPUs (same as HIP_VISIBLE_DEVICES=0,1). If the host has it set to empty, unset it: unset CUDA_VISIBLE_DEVICES. Do not pass --env CUDA_VISIBLE_DEVICES= (empty) into Docker -- that also hides all GPUs inside the container.

FP4BMM crash on gfx942 (MI300X) -- If the container exits immediately with a segfault or illegal instruction: VLLM_ROCM_USE_AITER_FP4BMM must be 0 on gfx942. This is set correctly in gpu_overrides.json for gfx942. See vLLM issue #34641.

HIP error: no kernel image -- The Docker image has no compiled kernel for your GPU's gfx version. Use vllm/vllm-openai-rocm:latest; it includes gfx942 and gfx950 kernels.

MLA models need --block-size 1 -- DeepSeek-R1/V3, Kimi-K2.5. Without it the MLA attention backend silently falls back to a slower path. This is in the recipe args for these models.

MoE models on multi-GPU need --distributed-executor-backend mp -- Qwen3-235B, GLM-4.5, MiniMax-M2. The default distributed executor does not work reliably with MoE on ROCm.

OOM during HIP graph capture -- If the container logs show "out of memory" after "capturing CUDA graphs" or "capturing HIP graphs", the model fits in VRAM but there isn't enough headroom for graph capture. Retry with --enforce-eager added to the vLLM args. This disables graph capture and frees 1-2 GB. Trade-off: slightly higher decode latency, but the model runs.

"Engine core initialization failed" -- This opaque error means the engine core subprocess died. Check early container logs: docker logs <name> 2>&1 | head -50. Common causes: gated model access denied (license not accepted on HF), unsupported architecture on this vLLM version, OOM during weight loading, missing --trust-remote-code for custom architectures, or vLLM version too old for the model (check min_vllm_version in the recipe).

/dev/kfd permission denied -- User is not in the video or render group. Fix: sudo usermod -aG video,render $USER (requires re-login).

SSH key not configured -- The scripts use BatchMode=yes SSH. If SSH fails with Permission denied (publickey), configure key-based access first.

Restricting GPUs on shared hosts -- Use --env HIP_VISIBLE_DEVICES=0,1 or --env CUDA_VISIBLE_DEVICES=0,1 to target specific GPUs by index. HIP_VISIBLE_DEVICES is the canonical AMD variable; CUDA_VISIBLE_DEVICES also works (ROCm maps it). Never set either to an empty string.

---

Reference

Precision compatibility, VRAM estimation, Docker flags, and known quirks: reference.md