This benchmark compares real SMG manual/min_load routing with Dynamo KV routing and Dynamo KV routing with explicit session affinity. New SMG sessions choose a worker with the fewest active requests; later requests reuse that worker. It uses the public SMG implementation, not a Python approximation or a static hash.
The experiment uses the open Qwen3-8B model. It isolates routing behavior that matters for g36 rolling: client-prepared token IDs, grouped branches, growing contexts, actual generated token IDs appended to subsequent requests, a two-second completion-relative turn gap, and immediate replacement of completed sessions. It does not reproduce the private g36 model or its exact prompt distribution.
Default Dynamo lost 23.0% and 22.5% throughput against real SMG load-aware stickiness. Session affinity restored cache locality, with throughput changes of -2.9% and -1.8%. Affinity did not beat SMG in either repeat. This public Qwen3-8B test reproduces the locality-loss pattern seen in the prior g36 study; it is not an executable copy of the private g36 workload.
Measured on 2026-09-14 with two GB300 GPUs, one BF16 Qwen3-8B TP1 worker per GPU, Dynamo 1.4.2, SGLang 0.5.16, and SMG 1.7.0. A runs SMG → KV → affinity; B reverses that order. Each trial runs for 180 seconds and scores backend output tokens during seconds 30–150.
| Routing | Output tokens/s A | Output tokens/s B | Change vs SMG A / B |
|---|---|---|---|
| SMG manual/min_load | 7,129 | 7,068 | +0.0% / +0.0% |
| Dynamo KV | 5,488 | 5,478 | -23.0% / -22.5% |
| Dynamo KV + affinity | 6,920 | 6,940 | -2.9% / -1.8% |
| Routing | Backend cache reuse A / B | Worker moves / successive turns A ; B | Uncached prefill tokens per output token A / B |
|---|---|---|---|
| SMG manual/min_load | 82.5% / 82.6% | 0/4,312 ; 0/4,267 | 4.36 / 4.34 |
| Dynamo KV | 54.2% / 54.0% | 1,485/3,324 ; 1,413/3,344 | 11.39 / 11.51 |
| Dynamo KV + affinity | 82.7% / 82.6% | 0/4,194 ; 0/4,227 | 4.35 / 4.44 |
All 27,322 requests / 6,994,432 generated tokens completed with no errors or cancellations. Independent checks matched backend/client request counts, input-token counts, and completed generation-token counts exactly; Dynamo trace cardinality also matched every completed session. All sticky trials had zero worker moves. Dynamo's warm-cache/flush gates passed before every trial.
The mechanism is visible in the counters: default routing moves growing contexts between workers and performs more uncached prefill work per output token. This configuration has no KV transfer between workers. Explicit affinity preserves the context's worker, restoring the locality the SMG baseline already provides. Inputs are token IDs, so there is no prompt-tokenization work for the Dynamo frontend to accelerate.
The earlier preempted diagnostic attempt is excluded. These six trials ran on one uninterrupted allocation; the container and its worker processes were verified inside that allocation's cgroup, and the job completed with zero restarts. Two repeats do not establish a confidence interval or rule out improvements from other Dynamo configurations or features.
results.json contains all six trials; metrics.json contains the sampled backend counters; requests.csv contains every completed request with worker placement, timing, counts, and token hashes. probes.json, validation.json, and provenance.json contain the protocol/cache checks, independent audits, settings, runtime versions, and hashes of the measured source. All six Python files were downloaded anonymously from this gist, matched the active run's source byte for byte, and passed the CPU contract tests. The pinned runtime manifest was also fetched anonymously and its digest verified.
Use two GB300 GPUs with at least 280 GB per GPU for the default cache allocation. The measured host is ARM64. The pinned digest below is a multi-platform image index with ARM64 and AMD64 variants; Docker selects the host architecture. Results on other architectures should be reported separately. Startup downloads about 16 GB of public model weights. No private code, data, credentials, or patched server is needed.
Clone this gist, change into its directory, then run:
git clone https://gist.github.com/f04f08087825a4e76643127d999b024e.git dynamo-rolling
cd dynamo-rolling
mkdir -p work
docker run --rm -it --gpus all --network host --ipc host \
-v "$PWD:/bench" -w /bench --entrypoint bash \
nvcr.io/nvidia/ai-dynamo/sglang-runtime@sha256:3692c0c6a1ae23045f48384a09b539dcb9347ffa66ead5d0426aa90e85865012
# Inside the container:
python3 -m venv work/smg-env
work/smg-env/bin/python -m pip install -r smg-requirements.txt
python3 check_smg.py --python "$PWD/work/smg-env/bin/python" --output work/check-smg
python3 check_rolling.py --output work/check-rolling
python3 rolling_bench.py prepare --work work
python3 rolling_bench.py run --work work --name rolling \
--smg-python "$PWD/work/smg-env/bin/python"
python3 summarize.py --root work/rolling --output work/shareablerun starts two SGLang workers, two Dynamo sidecars, a public KV event relay, and each gateway in sequence. It stops its child processes on exit. Run one copy at a time on a dedicated host: it uses HTTP 28000–28001 and 31000–31001, gRPC 32000–32001, and event ports 30000–30001. The CPU SMG fixture also uses 29010–29011. Output directories must be new.
The default run performs six 180-second trials in forward then reverse arm order, plus startup, probes, and bounded drain. Each trial scores the 120 seconds between time 30 and time 150. There are 256 active session slots, eight branches per group, twelve turns per session, 256 generated tokens per request, an eleven-second initial ramp, and a two-second gap after each completed request. The initial slots start at different turn depths. Later sessions start at turn zero.
Prompts use a deterministic pool of non-special tokens that individually decode to nonempty ASCII. The client loads the pinned tokenizer before serving begins. Sibling branches share a 512-token prefix; each branch then receives a distinct marker. Target context lengths grow quadratically from 1,024 to 16,384 tokens. Actual generated token IDs are inserted before each next-turn top-up. Outputs use greedy decoding with EOS disabled to fix request work. This synthetic shape is a routing mechanism test; it has no language-quality score.
Both workers use TP1, BF16, 1,048,576 KV token slots, 64 maximum running requests, 8,192-token prefill chunks, 16-token cache pages, and a 256-token stream interval. Every arm shares these engines and backend event publishing. This comparison does not measure the additional deployment cost of enabling event publishing on an otherwise event-free SGLang installation.
- SMG 1.7.0:
--policy manual --assignment-mode min_load, withX-SMG-Routing-Keyheld constant per session. The CPU fixture holds one worker busy, checks that a new session chooses the idle worker, then verifies that the old session stays on its busy worker. - Dynamo 1.4.2 KV:
--router-mode kv --router-kv-events, with a stableX-Dynamo-Session-IDbut no affinity setting. - Dynamo 1.4.2 sticky: the same configuration plus
--router-session-affinity-ttl-secs 3600.
SMG receives native /generate requests containing token IDs in a one-element batch. Dynamo receives /v1/completions requests containing token IDs and requests nvext.completion_token_ids. Both transports recover and validate every generated token ID. Before each trial, the routed greedy output must match direct SGLang exactly. Dynamo must also observe a prefix warmed directly on worker 1, then report zero overlap after a backend cache flush. The public KvEventPublisher relay binds each SGLang event stream to its sidecar's discovered worker ID. It implements live event forwarding; failover and snapshot recovery are outside this test.
The CPU rolling fixture checks the actual request bodies on both transports: each later prompt begins with the previous prompt plus the returned generated IDs. It also checks refill, mismatched output-count rejection, and that prefill counters are excluded from output throughput. Per-request routing traces must cover completed requests, and sticky sessions must have zero worker changes.
Output tokens/s is the sum of the two workers' sglang:realtime_tokens_total{mode="decode"} counter deltas divided by each worker's measured snapshot interval. This includes tokens from requests still running at the interval boundary. Startup, the first thirty seconds, and the final thirty seconds are excluded from scoring. No arrival schedule is replayed independently of response completion. The realtime and completed-generation counters are distinct: their full-trial totals differed by at most 55 tokens in these runs; both totals are published in validation.json. Exact client parity is checked against the completed-generation counter.
Backend cache reuse is 1 - uncached_prompt_tokens_histogram_sum / prompt_tokens_histogram_sum, using counter deltas in the same interior window. These histograms describe completed requests and can span a window boundary; they are diagnostic, not the throughput numerator. Queue depth, running requests, active KV fraction (sglang:token_usage), and free/evictable KV slot counts are sampled every two seconds. mean_kv_utilization reports the active fraction, not total cache residency. Worker movement is counted between successive completed requests of the same session. Requests still pending after the thirty-second drain deadline are explicitly cancelled and recorded; unexpected request errors abort the experiment.
Two reverse-order repeats do not provide a confidence interval. This test covers one model and a fixed two-worker topology on one host. It does not test cross-host communication, disaggregated prefill/decode, KV transfer or offload, autoscaling, fault recovery, multimodal input, or model weight updates. A result here cannot establish a universal Dynamo speedup or slowdown.
The existing g36 reference already uses load-aware initial placement followed by strict stickiness. Its SMG gateway selects manual policy with assignment_mode=min_load. The public SMG 1.7.0 manual policy implementation supplies that behavior in this benchmark, and the included busy-worker fixture tests it directly.
In the prior g36 study, each comparison shared two GB300 nodes, one TP4 g36 engine per node, across routers. At 3,072 active rolling sessions, default Dynamo KV routing reduced output token throughput by 28.1% and 29.3% in paired repeats. Backend cache reuse fell from about 75% to about 51–52%. Adding a 3,600-second affinity TTL restored roughly 75% cache reuse and zero worker moves; throughput changes versus the native reference were −1.9% and −0.1%. Those are prior private-model measurements, not new Qwen measurements or an executable g36 reproduction in this gist.
This public workload preserves the routing policy, causal context appends, grouped branches, completion-relative two-second gap, refill behavior, 256-token stream interval, and a 2:1 ratio of active session slots to backend running-request capacity. It substitutes a public 8B model, TP1 workers on one host, synthetic prompt lengths, greedy fixed-length outputs, and a smaller absolute workload. The results therefore test the locality mechanism; they do not establish numerical parity with g36 or prove that no other Dynamo feature or configuration can improve it.
The earlier finite Weka replay used static-hash stickiness. Its measured gains were real for that baseline but did not establish an improvement over the g36 reference, which already uses load-aware initial placement. Keep those results as a separate pilot; do not substitute them for this comparison.
Apache-2.0 for this harness; see LICENSE.txt. Model and runtime dependencies retain their own licenses.