Created
August 16, 2026 14:51
-
-
Save bigsnarfdude/d354a432d5fefbf9b07ddac4f6c50d7b to your computer and use it in GitHub Desktop.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Every 1.0s: nvidia-smi 192-222-51-153: Sun Aug 16 14:51:34 2026 | |
| Sun Aug 16 14:51:34 2026 | |
| +-----------------------------------------------------------------------------------------+ | |
| | NVIDIA-SMI 580.105.08 Driver Version: 580.105.08 CUDA Version: 13.0 | | |
| +-----------------------------------------+------------------------+----------------------+ | |
| | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | |
| | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | |
| | | | MIG M. | | |
| |=========================================+========================+======================| | |
| | 0 NVIDIA GH200 480GB On | 00000000:DD:00.0 Off | 0 | | |
| | N/A 34C P0 122W / 700W | 88842MiB / 97871MiB | 0% Default | | |
| | | | Disabled | | |
| +-----------------------------------------+------------------------+----------------------+ | |
| +-----------------------------------------------------------------------------------------+ | |
| | Processes: | | |
| | GPU GI CI PID Type Process name GPU Memory | | |
| | ID ID Usage | | |
| |=========================================================================================| | |
| | 0 N/A N/A 64382 C VLLM::EngineCore 88830MiB | | |
| +-----------------------------------------------------------------------------------------+ | |
Author
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The tokens/sec answer. vLLM BF16 + DFlash2 on GH200: 178 tok/s single-request, 1,030 tok/s
aggregate, 1,262 talks/hr on the real corrector. That's 4.6× against its own no-speculation
control — better than inco's published 2.7–3.4×, because your JSON output drafts unusually
well (acceptance 0.71–0.99). llama.cpp with the drafter only matches vLLM without it.
A build recipe that skips the 40-minute ARM source build — PR #52816 is pure Python/Triton,
so the released wheel's kernels plus the PR's Python gets you serving in ~15 minutes.