inkling-nvfp4-quickstart.md

Inkling NVFP4 — Quick Start Guide

Serving Thinking Machines Inkling (975B MoE, 41B active) with vLLM on 8x H200 GPUs.

Pre-requisites

• 8x H200 (or B200) GPUs
• ~552 GB disk for model weights
• Container runtime (podman or docker)

1. Download the model

HF_HUB_CACHE=/path/to/hub_cache \
  huggingface-cli download thinkingmachines/Inkling-NVFP4

2. Run the server

docker run -d --name vllm-inkling \
  --gpus all \
  --ipc=host \
  -p 8000:8000 \
  -v /path/to/hub_cache:/hf:ro \
  -e HF_HUB_CACHE=/hf \
  quay.io/vllm/automation-vllm:cuda-29455473553 \
  --model thinkingmachines/Inkling-NVFP4 \
  --tensor-parallel-size 8 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code \
  --kernel-config '{"enable_cutedsl_warmup": false}'

# With podman, drop --gpus and use --device nvidia.com/gpu=all instead

Note: HF_HUB_CACHE (not HF_HOME) — the model is stored directly in the cache dir, not under a hub/ subdirectory.

3. Smoke test

# Check model is loaded
curl http://localhost:8000/v1/models | python3 -m json.tool

# Chat completion
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "thinkingmachines/Inkling-NVFP4",
    "messages": [{"role": "user", "content": "What is Inkling?"}],
    "max_tokens": 256
  }'

Known issues (SM90 / H200)

The upstream PR (vllm-project/vllm#48768) targets SM100+ (Blackwell). Two patches were needed for H200 (SM90):

1. tml_fa4 flash attention — interface.py passes SM100-only args to cute.compile() for all arches. Patched via cmake post-fetch script to use SM90-compatible arg lists.
2. MoE gate ll_bf16 kernel — moe.py guards ll_bf16_gemm with has_device_capability(90), but CuTeDSL kernel compilation fails on SM90. Tightened to has_device_capability(100) so SM90 falls back to torch.mm.

These patches are baked into the image above. On B200 (SM100+), they're no-ops.

Image details

Field	Value
Image	quay.io/vllm/automation-vllm:cuda-29455473553
Base	nm-vllm-ent branch doug/inkling-0day (sync-v0.25.0 + PR #48768)
CUDA	13.0
Python	3.12
VRAM per GPU	~131 GB (70 GB weights + KV cache)
KV cache	158K tokens (~19x concurrent 8K requests)