Gwen (Qwen3.6-27B PrismaAURA 5.5bit CT, vLLM 0.26.0, 2x RTX 3090 TP=2) vLLM config A/B: original config + spec-tokens variants with measured throughput

gwen-original-config.service

[Unit]
Description=vLLM OpenAI-compatible server (Qwen3.6-27B PrismaAURA 5.5bit CT, TP=2, MTP spec=5)
After=network.target
Wants=network.target

[Service]
Type=simple
User=alex
Group=alex
ExecStart=/home/alex/vllm-venv/bin/python -m vllm.entrypoints.openai.api_server \
  --model /home/alex/.cache/huggingface/hub/models--rdtand--Qwen3.6-27B-PrismaAURA-5.5bit-vllm/snapshots/d017bf96ad8214e18fa6d9bbed3e9dfbdba6309f \
  --served-model-name qwen3-27b \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.97 \
  --max-model-len 98304 \
  --max-num-seqs 128 \
  --quantization compressed-tensors \
  --dtype bfloat16 \
  --enable-prefix-caching \
  --kv-cache-dtype fp8 \
  --spec-method qwen3_5_mtp \
  --spec-model /home/alex/.cache/huggingface/hub/models--rdtand--Qwen3.6-27B-PrismaAURA-5.5bit-vllm/snapshots/d017bf96ad8214e18fa6d9bbed3e9dfbdba6309f \
  --spec-tokens 3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --default-chat-template-kwargs '{"enable_thinking": false}' \
  --host 0.0.0.0 --port 8080
Restart=on-failure
RestartSec=10
TimeoutStartSec=600
StandardOutput=journal
StandardError=journal
SyslogIdentifier=vllm-qwen
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Variant: spec-tokens 3 (ORIGINAL production config)

This is the config that was running in production. --spec-tokens 3, self-speculative MTP (draft model = same checkpoint, embedding/lm_head weights shared).

Config:

--spec-method qwen3_5_mtp
--spec-model  /home/alex/.cache/huggingface/hub/models--rdtand--Qwen3.6-27B-PrismaAURA-5.5bit-vllm/snapshots/d017bf96ad8214e18fa6d9bbed3e9dfbdba6309f
--spec-tokens 3

Measured (warm graph cache, session-start probe):

concurrency	aggregate tok/s	per-stream avg
1	46.7	46.8
2	99.2	51.9
4	93.1	24.2
8	208.6	30.0

⚠️ CRITICAL FINDING: OOM on cold boot. After any restart that invalidates the CUDA-graph compile cache, the FIRST request dies in graph capture:
RuntimeError: Triton Error [CUDA]: out of memory at vllm/compilation/cuda_graph.py:254 → EngineCore encountered a fatal error → service deactivates. Reproduced twice in a row on clean boots. Root cause: --gpu-memory-utilization 0.97 leaves ~500MB headroom; the 3-token MTP spec buffers + PIECEWISE graph capture tip it over. The 46.7 tok/s measurement only worked because the previous boot had a warm graph cache.

Variant: spec-tokens 1 (WINNER — current production config)

Single-token MTP speculation. Same method/model, just 1 spec token. This avoids the CUDA-graph capture OOM entirely (tiny buffer footprint) while keeping ~97% of the spec win.

Config:

--spec-method qwen3_5_mtp
--spec-model  /home/alex/.cache/huggingface/hub/models--rdtand--Qwen3.6-27B-PrismaAURA-5.5bit-vllm/snapshots/d017bf96ad8214e18fa6d9bbed3e9dfbdba6309f
--spec-tokens 1

Measured (2 clean boots, reproduced):

concurrency	aggregate tok/s	per-stream avg
1	75.9	75.9
2	103.2	53.5
4	189.7	47.7
8	232.3	29.3

Acceptance: 1891/1950 = 97% (spec_decode_num_accepted_tokens / num_draft_tokens, per-position pos=0).

Why it wins:

• Single-stream 75.9 vs 36.6 (no-spec) = +107% — above the ~100 tok/s bandwidth wall, MTP minting >1 token per weight-read
• vs the fragile spec=3: +62% single-stream, AND stable across restarts (no graph-capture OOM)
• Aggregate at 8 concurrency (232.3) ≈ no-spec (233.1) — spec overhead at high batch is negligible; the win concentrates at low concurrency where per-stream latency matters (agent/review workloads)

Variant: spec-tokens 5 (REJECTED)

Config:

--spec-method qwen3_5_mtp
--spec-model  <same checkpoint>
--spec-tokens 5

Measured:

concurrency	aggregate tok/s	per-stream avg
1	44.1	44.1
2	41.2	21.7
4	10.7	2.7
8	—	HTTP 500

Verdict: reject. Performance COLLAPSES as concurrency rises (10.7 aggregate at 4 = 4x worse than single-stream) then 500s. Signature of memory thrash: 5-token spec buffers + graph capture exceed the ~500MB headroom at gpu-mem-util 0.97. Worse than both spec=1 AND no-spec — higher spec tokens is not monotonic on this memory-constrained box.

Variant: spec-tokens 8 (REJECTED — non-reproducible)

Config:

--spec-method qwen3_5_mtp
--spec-model  <same checkpoint>
--spec-tokens 8

Measured (boot 1):

concurrency	aggregate tok/s	per-stream avg
1	124.8	124.9
2	90.9	55.7
4	200.7	50.2
8	—	HTTP 500

Measured (boot 2, same config):

concurrency	aggregate tok/s	per-stream avg
1	46.8	46.8
2	89.5	52.4
4	64.4	16.1

Verdict: reject. The 124.8 single-stream headline does NOT reproduce — boot 2 gives 46.8 for the identical config. 3x variance between boots = memory thrash, not a config effect. 500s at 8 concurrency. Whatever the true number is, it's unstable enough that it cannot be trusted in production.

⚠️ MEASUREMENT METHODOLOGY CORRECTION (all numbers above)

Two measurement distortions were discovered after the original A/B ran — read this before trusting any absolute tok/s in the comments above.

1. My word-count probe UNDER-counts real tokens. The probe counted whitespace-separated words in streamed deltas, not tokens. English ≈ 0.7 words/token, so every "tok/s" in the A/B tables is ~25-30% LOW. Example: the "245 tok/s @8" for spec-tokens 1 is really ~340 real tokens/s (measured via vLLM's generation_tokens_total counter delta over a timed run). Relative comparisons between configs are still valid; absolute numbers are not.

2. vLLM's logger "Avg generation throughput" OVER-counts. The engine log line sums drafted + accepted speculative tokens, not real output. Drafts are proposals (often discarded), so the logger number is inflated. Verified: at the 368.9 peak, Accepted throughput: 179.11 + Drafted throughput: 189.21 = 368.32. Any "400 tok/s" seen in vLLM logs is drafted+accepted, not real output.

Corrected real throughput (spec-tokens 1, 2x RTX 3090 TP=2, current config):

concurrency	word-probe (undercounts)	REAL generation tok/s (counter delta)	vLLM logger (overcounts, drafts+accepted)
1	75.9	66.9	varies, up to ~368 peak
8	245	339.9	~368 peak

Takeaway: real output at 8 concurrent streams ≈ 340 tok/s, both GPUs at 98% util — that's the physical bandwidth wall for 18.6 GB weights / 1.87 TB/s HBM. To genuinely exceed it needs lower-bit weights or more bandwidth, not config. The spec-tokens 1 win (+107% single-stream over no-spec) and the OOM findings are unaffected by this correction — those were relative/behavioral, not absolute-throughput claims.

Measurement method that works: read vllm:generation_tokens_total from /metrics, run the load, read it again, divide by wall time. Trust that over word-counting or the logger's generation line.