Inkling (975B param, 41B active MoE) serves on 8x H200 using the thinkingmachines/Inkling-NVFP4 checkpoint. The upstream PRs target Blackwell (SM100), so three patches are needed for Hopper (SM90).
All patches live on doug/inkling-0day (based on the upstream Add Inkling multi-depth MTP support [5/N] PR merged onto our v0.25.0 base).
| Commit | Description |
|---|---|
9d07a68 |
fix(inkling): require SM100+ for CuTeDSL ll_bf16 gate kernel — tightens the ll_bf16_gemm guard from has_device_capability(90) to has_device_capability(100) so SM90 falls back to torch.mm instead of hitting a CuTeDSL MLIR compilation crash |
8868310 |
[Inkling] Use unused token ids for image/audio placeholders — cherry-pick of upstream #48775 |
e9a69be |
Fix tml_fa4 SM90 compat: split-KV guard + compile call args — cmake post-fetch patch on tml_fa4/interface.py that guards SM100-specific cute.compile() args and clamps num_splits=1 on SM90 |
_flash_attn_fwd passes SM100-specific positional args (logits_max_tensor, sfq_tensor, etc.) to cute.compile() for ALL architectures. The SM90 kernel has a smaller __call__ signature, so it crashes with DSLRuntimeError: too many positional arguments. The split-KV path (num_splits=32) also asserts on SM90.
Fix: Arch check so only SM100/SM110 passes the extended arg list. SM90 gets a reduced compile/exec call. num_splits clamped to 1 on capability[0] < 10. Applied as a cmake post-fetch patch.
_linear_with_fp32_out() guards the CuTeDSL path with has_device_capability(90), which passes on H200. But CuTeDSL kernel compilation fails on SM90 with a ThreadIdXOp MLIR binding error. The torch.mm fallback works fine.
Fix: Tightened guard to has_device_capability(100).
CuTeDSL warmup (inkling_fa4 compile units) hangs on SM90 because the patched interface doesn't support the warmup codepath. Disabled at launch via --kernel-config.
podman run -d --name vllm-inkling \
--device nvidia.com/gpu=all \
--ipc=host \
--security-opt=label=disable \
-p 8000:8000 \
-v $HOME/.cache/huggingface/hub:/hf:ro \
-e HF_HUB_CACHE=/hf \
-e HF_HUB_OFFLINE=1 \
quay.io/vllm/automation-vllm:cuda-29455473553 \
--model thinkingmachines/Inkling-NVFP4 \
--tensor-parallel-size 8 \
--max-model-len 8192 \
--gpu-memory-utilization 0.90 \
--trust-remote-code \
--host 0.0.0.0 --port 8000Note: On B200 (SM100/Blackwell), none of these patches are needed — NVFP4 is native. Add the flashinfer cubins mount for the SM100 cubin workaround.
- VRAM: ~131 GB per GPU (70 GB weights + KV cache)
- KV cache: 158K tokens (~19x concurrent 8K requests)
- Startup: ~10 min (weight load + autotuning, no CuTeDSL warmup)
- Output quality: TBD — NVFP4 quant on SM90 uses dequant-to-higher-precision paths, not native FP4 tensor cores
We reported these findings on the upstream PR:
Inkling-NVFP4 runs natively on 8x B200 (SM100, Blackwell) — no patches needed, FP4 tensor cores used directly. See the B200 demo gist for example queries (text + multimodal).