benchmark-results.md

Benchmark results — Nemotron-3-Super-120B-A12B-NVFP4 on DGX Spark / GB10

Hardware: Dell Pro Max GB10 (DGX Spark class, Grace-Blackwell sm_121, ~121.6 GiB unified LPDDR5X). Server: vLLM cu130-nightly (0.19.2rc1.dev134), NVFP4 weights, MoE on Marlin, attention TRITON_ATTN, KV fp8, single GPU (TP/PP/DP=1). Load tool: guidellm 0.6.0, concurrent profile, chat shape (prompt≈512 / output≈256), 180 s per concurrency level. Metric = output (decode) tokens/sec, mean of successful requests; "engine peak" = vLLM's own Avg generation throughput log at saturation.

1. Single-stream MTP tuning (num_speculative_tokens, temp 1.0 / top_p 0.95)

The MTP head reuses a single draft layer per position, so acceptance decays fast at the real serving temperature:

draft position	acceptance @ temp 1.0
pos 0 (1st draft)	~67%
pos 1 (2nd draft)	~36%
pos 2 (3rd draft)	~16%

config	single-stream tok/s (temp 1.0)
no MTP (baseline)	~15–17
MTP k=3 (NVIDIA guide default)	~23.5
MTP k=2 (chosen)	~24.2

k=2 is as fast as k=3 and cheaper (one fewer draft forward/step, less draft KV → better under concurrency). The 3rd draft token (16% accepted) doesn't earn its forward. Note: greedy/temp=0 reads ~25 tok/s because deterministic sampling inflates acceptance; use the temp-1.0 number for real expectations.

2. max_num_seqs sweep (aggregate throughput, MTP k=2, --max-num-batched-tokens 8192)

Saturated aggregate output throughput vs the server-side concurrency cap:

max_num_seqs	sat. out tok/s	engine peak	KV @ sat	TTFT @ sat	step
4 (baseline, sched-tokens 2048)	47.7	~50	7.6%	1.2 s	—
8	73.5	~82	16.6%	6.6 s	+54%
16	99.6	~117	33.1%	7.7 s	+36%
32	131.3	176	68.9%	9.1 s	+32%

~2.75× aggregate throughput from raising max_num_seqs alone (47.7 → 131 tok/s).

Key reads:

• The bottleneck at the default max_num_seqs=4 was the scheduler cap, not GPU resources — KV cache sat at 7.6% with the GPU ~92% idle on memory.
• We are KV-bound, not compute-bound, at 32: engine peak still grew 1.5× from 16→32, but KV is at 69%, so maxseq=48 (~103% KV) won't fit at this context length. 32 is the practical ceiling for the chat shape. Longer contexts (e.g. coder 4k+) bind KV much sooner (~5–6 concurrent).
• Cost is latency: TTFT climbs to ~9 s at saturation — fine for batch/async fan-out, not for snappy interactive use. Pick max_num_seqs to your latency SLO, not the max.

Per-level detail (out tok/s by client concurrency)

client conc →	1	4	8	16	32	64
maxseq=4	23.9	47.7	46.1	45.8	45.7	—
maxseq=8	24.3	46.3	73.5	71.8	71.9	—
maxseq=16	23.9	48.0	74.7	99.6	91.4	—
maxseq=32	23.6	47.1	67.0	99.4	131.3	136.3

Throughput flattens once client concurrency exceeds the server's max_num_seqs (extra requests queue → TTFT rises, throughput flat). Bold = each config's saturation point.

3. MTP on vs off, under concurrency

Does pure batching beat batch+speculation once the GPU is saturated? No — MTP wins.

config	sustained tok/s	engine peak	KV @ sat	single-stream	TTFT @ sat
MTP on, maxseq=32	136	176	69%	24	9 s
MTP off, maxseq=32	96–109	134	18%	18	12 s
MTP off, maxseq=64	121	192	36%	18	15 s

• The GB10 is memory-bandwidth-bound on decode, so MTP's "more tokens per weight-load" advantage survives batching — MTP-on sustains more than MTP-off even at full occupancy.
• MTP-off briefly peaks higher at maxseq=64 (192 vs 176) thanks to 3.8× better KV efficiency (MTP-on burns 2.15%/seq vs 0.57%/seq for its draft/lookahead slots — which is why MTP-on is KV-capped at ~32), but its sustained throughput is lower and latency is far worse. Not worth it.
• Verdict: keep MTP on (k=2). Best single-stream AND best sustained aggregate throughput.

4. Prefix caching — does NOT work on this model (negative result)

Enabled --enable-prefix-caching (vLLM auto-sets Mamba cache mode 'all', flagged experimental). Drove a real shared-prefix load: identical ~900-token system prompt (SYSTEM.md) + varied user questions, then the exact same request repeated 3×.

metric	result
vllm:prefix_cache_hits_total	0 (across 9,317 queries)
engine Prefix cache hit rate	0.0%
TTFT, identical request repeated 3×	flat ~690 ms (no improvement)

Zero cache hits, even for byte-identical repeated prompts. Hybrid-Mamba prefix caching is non-functional/ineffective for Nemotron-3-Super on this nightly — it boots without crashing but provides no reuse. Not a usable lever today. (Watch for it to mature; the recurrent Mamba state is the hard part to prefix-share.)

Final recommendation

• Throughput-max (batch / agent fan-out via LiteLLM): MTP k=2 + --max-num-seqs 32
  • --max-num-batched-tokens 8192 → ~136 tok/s sustained (2.85× the original baseline), TTFT ~9 s under load.
• Latency-sensitive (interactive): MTP k=2 + --max-num-seqs 4 → ~24 tok/s single-stream, ~47 tok/s aggregate, TTFT ~1.2 s.
• Prefix caching: skip (0% hits). MTP-off: skip (MTP-on is strictly better here).

docker-vllm-nemotron-3-super-120b-nvfp4-mtp-throughput.sh

#!/usr/bin/bash
#
# THROUGHPUT profile: identical to the MTP script but tuned for AGGREGATE tokens/sec
# under concurrent load (batch / agent fan-out via LiteLLM), not single-stream latency.
# Deltas vs ...-MTP.sh:  --max-num-seqs 4 -> 32,  + --max-num-batched-tokens 8192.
# Measured 2026-06-09 (chat 512/256): ~136 tok/s sustained (~2.85x the maxseq=4 config),
# at the cost of TTFT ~9 s under saturation. KV-bound at ~32 for chat-length contexts;
# drop max-num-seqs for long contexts or tighter latency. See docs/02 runbook (F2).
#
# Serve NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 with vLLM on the GB10, WITH the
# built-in MTP speculative-decoding head enabled.
#
# This is the throughput-tuned variant of
# docker-vllm-RedHatAI-NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4.sh. It follows
# NVIDIA's official Nemotron-3-Super DGX-Spark deployment guide:
#   https://docs.nvidia.com/nemotron/nightly/usage-cookbook/Nemotron-3-Super/SparkDeploymentGuide/
#
# WHY A DIFFERENT IMAGE: the MTP head has its own *unquantized* MoE layer. The
# NVFP4 main MoE requires --moe-backend marlin on sm_121, but marlin is invalid
# for the unquantized MTP MoE -- and v0.17.1 has only ONE global --moe-backend, so
# the two cannot be satisfied at once (startup ValueError). The fix is a per-path
# override: a nested "moe_backend":"triton" INSIDE --speculative_config, routing
# the MTP head to triton while the main MoE stays on marlin. That nested key does
# NOT exist in v0.17.1's SpeculativeConfig; it requires the cu130-nightly track.
#
# IMAGE PIN: cu130-nightly is a ROLLING tag (NVIDIA: "compatibility track, not a
# reproducible pin"). Tested-good digest below. Some nightlies (e.g. 473de04c...)
# crash with a CUDA illegal-memory-access on the FIRST MTP request -- if that
# happens, pull a newer nightly. Re-pin the digest after any successful upgrade.
#   vllm 0.19.2rc1.dev134+gfe9c3d6c5
#   vllm/vllm-openai@sha256:3dbe092ec5b2cef63b6104d33fa75d6ce53a7870962529ada69f78bbbc38e776
#
# DELTAS vs the known-good v0.17.1 script (everything else is identical):
#   - image            v0.17.1-cu130            -> cu130-nightly
#   - VLLM_USE_FLASHINFER_MOE_FP4=0  (NEW)       FlashInfer FP4 MoE is Blackwell
#                                                multi-GPU only; force Marlin on Spark.
#                                                Also stops v0.19+ auto-selecting
#                                                FLASHINFER_CUTLASS over Marlin.
#   - --gpu-memory-utilization 0.82  -> 0.90     per the guide (MTP needs headroom;
#                                                if it OOMs, lower this first).
#   - --speculative_config ...        (NEW)       MTP head, num_speculative_tokens 2,
#                                                MoE backend triton (per-path override).
#
# NUM_SPECULATIVE_TOKENS=2 (not the guide's 3): A/B on 2026-06-09 at the real
# serving temp (1.0/top_p 0.95) gave k=2 ~24.2 tok/s vs k=3 ~23.5 (same prompt).
# The single MTP layer is reused per draft position, so acceptance decays fast at
# temp 1.0 (pos0 ~67% -> pos1 ~36% -> pos2 ~16%); the 3rd draft token barely pays
# for its forward. k=2 is as fast and cheaper (less draft compute/KV -> better
# under concurrency). Re-check per-position acceptance via /metrics if the typical
# workload shifts toward greedy/code (where higher k wins). See docs/02 (F1).
#
# SINGLE-HOST: stop the other vLLM containers first (free >= util x total at start).
source ../.env
export VLLM_IMAGE="vllm/vllm-openai:cu130-nightly"          # guide's DGX-Spark MTP image
export MODEL_DIR=RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
export SERVED_NAME="${MODEL_DIR}"
export CONTAINER_NAME=VLLM-NEMOTRON-3-SUPER-120B-A12B-NVFP4
export MAX_MODEL_LEN=262144

docker run -d --name "${CONTAINER_NAME}" \
	--gpus all \
	--ipc=host \
	-p 8000:8000 \
	-v ~/.cache/huggingface:/root/.cache/huggingface \
	--env "HF_TOKEN=$HF_TOKEN" \
	-e TOKENIZERS_PARALLELISM=false \
	-e VLLM_NVFP4_GEMM_BACKEND=marlin \
	-e VLLM_USE_FLASHINFER_MOE_FP4=0 \
	-e VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm \
	"${VLLM_IMAGE}" \
		"${MODEL_DIR}" \
		--served-model-name "${SERVED_NAME}" \
		--trust-remote-code \
		--dtype auto \
		--tensor-parallel-size 1 \
		--pipeline-parallel-size 1 \
		--data-parallel-size 1 \
		--max-model-len "${MAX_MODEL_LEN}" \
		--max-num-seqs 32 \
		--max-num-batched-tokens 8192 \
		--gpu-memory-utilization 0.90 \
		--kv-cache-dtype fp8 \
		--mamba-ssm-cache-dtype float32 \
		--attention-backend TRITON_ATTN \
		--moe-backend marlin \
		--async-scheduling \
		--enable-chunked-prefill \
		--max-cudagraph-capture-size 128 \
		--speculative_config '{"method":"mtp","num_speculative_tokens":2,"moe_backend":"triton"}' \
		--reasoning-parser nemotron_v3 \
		--enable-auto-tool-choice \
		--tool-call-parser qwen3_coder \
		--override-generation-config '{"temperature":1.0,"top_p":0.95}'

docker-vllm-nemotron-3-super-120b-nvfp4-mtp.sh

#!/usr/bin/bash
#
# Serve NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 with vLLM on the GB10, WITH the
# built-in MTP speculative-decoding head enabled for higher single-stream tok/s.
#
# This is the MTP-enabled variant of
# docker-vllm-RedHatAI-NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4.sh. It follows
# NVIDIA's official Nemotron-3-Super DGX-Spark deployment guide:
#   https://docs.nvidia.com/nemotron/nightly/usage-cookbook/Nemotron-3-Super/SparkDeploymentGuide/
#
# WHY A DIFFERENT IMAGE: the MTP head has its own *unquantized* MoE layer. The
# NVFP4 main MoE requires --moe-backend marlin on sm_121, but marlin is invalid
# for the unquantized MTP MoE -- and v0.17.1 has only ONE global --moe-backend, so
# the two cannot be satisfied at once (startup ValueError). The fix is a per-path
# override: a nested "moe_backend":"triton" INSIDE --speculative_config, routing
# the MTP head to triton while the main MoE stays on marlin. That nested key does
# NOT exist in v0.17.1's SpeculativeConfig; it requires the cu130-nightly track.
#
# IMAGE PIN: cu130-nightly is a ROLLING tag (NVIDIA: "compatibility track, not a
# reproducible pin"). Tested-good digest below. Some nightlies (e.g. 473de04c...)
# crash with a CUDA illegal-memory-access on the FIRST MTP request -- if that
# happens, pull a newer nightly. Re-pin the digest after any successful upgrade.
#   vllm 0.19.2rc1.dev134+gfe9c3d6c5
#   vllm/vllm-openai@sha256:3dbe092ec5b2cef63b6104d33fa75d6ce53a7870962529ada69f78bbbc38e776
#
# DELTAS vs the known-good v0.17.1 script (everything else is identical):
#   - image            v0.17.1-cu130            -> cu130-nightly
#   - VLLM_USE_FLASHINFER_MOE_FP4=0  (NEW)       FlashInfer FP4 MoE is Blackwell
#                                                multi-GPU only; force Marlin on Spark.
#                                                Also stops v0.19+ auto-selecting
#                                                FLASHINFER_CUTLASS over Marlin.
#   - --gpu-memory-utilization 0.82  -> 0.90     per the guide (MTP needs headroom;
#                                                if it OOMs, lower this first).
#   - --speculative_config ...        (NEW)       MTP head, num_speculative_tokens 2,
#                                                MoE backend triton (per-path override).
#
# NUM_SPECULATIVE_TOKENS=2 (not the guide's 3): A/B on 2026-06-09 at the real
# serving temp (1.0/top_p 0.95) gave k=2 ~24.2 tok/s vs k=3 ~23.5 (same prompt).
# The single MTP layer is reused per draft position, so acceptance decays fast at
# temp 1.0 (pos0 ~67% -> pos1 ~36% -> pos2 ~16%); the 3rd draft token barely pays
# for its forward. k=2 is as fast and cheaper (less draft compute/KV -> better
# under concurrency). Re-check per-position acceptance via /metrics if the typical
# workload shifts toward greedy/code (where higher k wins). See docs/02 (F1).
#
# SINGLE-HOST: stop the other vLLM containers first (free >= util x total at start).
source ../.env
export VLLM_IMAGE="vllm/vllm-openai:cu130-nightly"          # guide's DGX-Spark MTP image
export MODEL_DIR=RedHatAI/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
export SERVED_NAME="${MODEL_DIR}"
export CONTAINER_NAME=VLLM-NEMOTRON-3-SUPER-120B-A12B-NVFP4
export MAX_MODEL_LEN=262144

docker run -d --name "${CONTAINER_NAME}" \
	--gpus all \
	--ipc=host \
	-p 8000:8000 \
	-v ~/.cache/huggingface:/root/.cache/huggingface \
	--env "HF_TOKEN=$HF_TOKEN" \
	-e TOKENIZERS_PARALLELISM=false \
	-e VLLM_NVFP4_GEMM_BACKEND=marlin \
	-e VLLM_USE_FLASHINFER_MOE_FP4=0 \
	-e VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm \
	"${VLLM_IMAGE}" \
		"${MODEL_DIR}" \
		--served-model-name "${SERVED_NAME}" \
		--trust-remote-code \
		--dtype auto \
		--tensor-parallel-size 1 \
		--pipeline-parallel-size 1 \
		--data-parallel-size 1 \
		--max-model-len "${MAX_MODEL_LEN}" \
		--max-num-seqs 4 \
		--gpu-memory-utilization 0.90 \
		--kv-cache-dtype fp8 \
		--mamba-ssm-cache-dtype float32 \
		--attention-backend TRITON_ATTN \
		--moe-backend marlin \
		--async-scheduling \
		--enable-chunked-prefill \
		--max-cudagraph-capture-size 128 \
		--speculative_config '{"method":"mtp","num_speculative_tokens":2,"moe_backend":"triton"}' \
		--reasoning-parser nemotron_v3 \
		--enable-auto-tool-choice \
		--tool-call-parser qwen3_coder \
		--override-generation-config '{"temperature":1.0,"top_p":0.95}'

findings.md

Maximizing vLLM throughput for Nemotron-3-Super-120B-A12B-NVFP4 on a DGX Spark / GB10

Notes from getting nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 (RedHatAI repack) to run fast on a single Dell Pro Max GB10 (DGX Spark class, Grace-Blackwell sm_121, ~121.6 GiB unified memory) with vLLM. Three things mattered: getting MTP speculative decoding to run at all, tuning it, and then unlocking aggregate throughput with batching.

Model: 120B total / 12B active, hybrid Mamba-2 + Latent-MoE + attention with a built-in MTP (multi-token prediction) head, ~75 GiB of NVFP4 weights.

---

1. MTP + Marlin is a hard conflict on vLLM 0.17.1 — and the fix isn't a version bump

The model card's DGX-Spark recipe pins vllm/vllm-openai:v0.17.1-cu130 with --moe-backend marlin (Marlin is the only working NVFP4 MoE path on sm_121 — the CUTLASS/FlashInfer grouped-FP4 kernels are gated to sm_100 or regress on sm_120/121). That serves fine without MTP.

Turn MTP on (--speculative-config '{"method":"mtp","num_speculative_tokens":2}') and it crashes at startup:

ValueError: moe_backend='marlin' is not supported for unquantized MoE.
Expected one of ['triton', 'flashinfer_trtllm', 'flashinfer_cutlass', 'aiter'].
  (vllm/model_executor/models/nemotron_h_mtp.py → select_unquantized_moe_backend)

Root cause: the MTP head has its own MoE layer that is unquantized (not NVFP4). vLLM 0.17.1 has a single global --moe-backend fed to both the NVFP4 main-MoE oracle (needs marlin) and the unquantized MTP-MoE oracle (rejects marlin). One flag can't satisfy both, and 0.17.1's SpeculativeConfig has no per-path override.

Fix (per NVIDIA's official Nemotron-3-Super Spark deployment guide): a per-path MoE backend override nested inside the speculative config — keep the main MoE on Marlin, route the MTP head to Triton:

--moe-backend marlin \
--speculative_config '{"method":"mtp","num_speculative_tokens":2,"moe_backend":"triton"}'

The inner moe_backend key does not exist in v0.17.1 — it landed in the cu130-nightly track. So enabling MTP is an image change, not just a flag. We run vllm/vllm-openai:cu130-nightly (vLLM 0.19.2rc1.dev134, digest sha256:3dbe092e…).

Verify in the startup log — you want both:

Using 'MARLIN' NvFp4 MoE backend         (main MoE,  nvfp4.py)
Using TRITON Unquantized MoE backend      (MTP head, unquantized.py)

Gotchas

• Don't "just drop --moe-backend" (auto). Since v0.19.0 NVFP4 MoE auto-selects FLASHINFER_CUTLASS, not Marlin, on sm_120-class. The explicit --moe-backend marlin is still required; keep VLLM_USE_FLASHINFER_MOE_FP4=0 so FlashInfer FP4 MoE (Blackwell multi-GPU only) doesn't get auto-picked.
• Keep --attention-backend TRITON_ATTN — FlashInfer + MTP crashes on sm_121 (vLLM #37754).
• cu130-nightly is a rolling tag. Some nightly digests crash with a CUDA illegal-memory-access on the first MTP request. Pin a known-good digest; keep the pinned v0.17.1/no-MTP script as a fallback.

2. num_speculative_tokens=2, not the guide's 3

The MTP head reuses one draft layer per position, so acceptance decays steeply at the real serving temperature (1.0 / top_p 0.95): pos0 ~67% → pos1 ~36% → pos2 ~16%. The 3rd draft token rarely earns its forward pass. A same-prompt A/B gave k=2 ~24.2 tok/s vs k=3 ~23.5, and k=2 is cheaper (less draft compute/KV). Re-check via /metrics (vllm:spec_decode_num_accepted_tokens_per_pos_total) if your workload skews greedy/code, where higher k wins. MTP took single-stream from ~15–17 → 24 tok/s (**+45%**).

3. The single biggest aggregate win: raise max_num_seqs

The default max_num_seqs=4 left the GPU ~92% idle on KV (7.6% cache used). Raising the scheduler cap (with --max-num-batched-tokens 8192) scaled aggregate decode throughput ~2.75× — 47.7 → 73.5 → 99.6 → 131 tok/s at maxseq 4 → 8 → 16 → 32 (engine peak 176). At 32 we become KV-bound (69% at chat-length context), not compute-bound — so 32 is the practical ceiling for short-context chat; long contexts bind much sooner. The trade is TTFT (1.2 s → ~9 s at saturation), so pick max_num_seqs to your latency SLO. Full tables in benchmark-results.md.

4. MTP still wins under heavy concurrency (don't disable it for batching)

Intuition says speculation should hurt once the batch is full (draft compute competes with real work). Measured the opposite: at maxseq=32, MTP-on sustains ~136 tok/s vs MTP-off's ~96–109. The GB10 is memory-bandwidth-bound, so MTP's extra tokens-per-weight-load keeps paying off even batched. MTP-off does use 3.8× less KV per sequence (0.57% vs 2.15%), so it can pack more sequences — at maxseq=64 it briefly peaks at 192 vs MTP-on's 176 — but its sustained throughput is still lower and latency much worse. Keep MTP on (k=2).

5. Prefix caching does not work on this model (negative result)

--enable-prefix-caching boots (vLLM auto-enables experimental "Mamba cache mode 'all'") but yields 0 cache hits — even byte-identical requests repeated back-to-back show prefix_cache_hits_total=0, engine hit rate 0.0%, and flat TTFT. Hybrid-Mamba prefix caching is non-functional/ineffective for Nemotron-3-Super here (the recurrent Mamba state is the hard part to share). Not a usable lever today.

TL;DR recommended config

Common base (all profiles):

• Image vllm/vllm-openai:cu130-nightly (pin the digest)
• --moe-backend marlin + env VLLM_NVFP4_GEMM_BACKEND=marlin, VLLM_USE_FLASHINFER_MOE_FP4=0, VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm
• --attention-backend TRITON_ATTN, --kv-cache-dtype fp8, --mamba-ssm-cache-dtype float32
• --speculative_config '{"method":"mtp","num_speculative_tokens":2,"moe_backend":"triton"}'
• --gpu-memory-utilization 0.90

Then pick by workload:

• Throughput-max (batch / agent fan-out): --max-num-seqs 32 --max-num-batched-tokens 8192 → ~136 tok/s sustained (2.85× baseline), TTFT ~9 s under load.
• Latency-sensitive (interactive): --max-num-seqs 4 → ~24 tok/s single-stream, ~47 tok/s aggregate, TTFT ~1.2 s.

Skip prefix caching (0% hits on this arch) and don't disable MTP for batching (MTP-on wins).

Full launch script: docker-vllm-nemotron-3-super-120b-nvfp4-mtp.sh. Fronted by LiteLLM: litellm_config.yaml.

References

• NVIDIA Nemotron-3-Super DGX Spark deployment guide — https://docs.nvidia.com/nemotron/nightly/usage-cookbook/Nemotron-3-Super/SparkDeploymentGuide/
• vLLM MTP docs — https://docs.vllm.ai/en/latest/features/speculative_decoding/mtp/
• NVFP4 MoE auto-select / no override on SM120 — vLLM #38971
• FlashInfer + MTP crash on sm_121 — vLLM #37754
• Mamba prefix-caching + MTP crash (NemotronH) — vLLM #39809
• b12x FP4 MoE for SM120/121 — vLLM PR #40082, #43328

Measured 2026-06-09. cu130-nightly moves; re-validate periodically.