Endpoint: gx10 120B (Qwen 122B) at http://gx10-83fb.tail3dac72.ts.net:8002
Script: scripts/benchmark_120b_tokens_per_second.py
| What we measure | How |
|---|---|
| Output tokens/s | Non-streaming completion; usage.completion_tokens / wall_time |
| TTFT (time to first token) | Optional streaming run; time until first content chunk |
| Variance | Multiple runs (--runs N); mean ± std |
| Concurrency | Parallel requests (--concurrent C); aggregate throughput |
See script help: python3 scripts/benchmark_120b_tokens_per_second.py --help
The following ranges are realistic for self-hosted 70B–120B (not vendor fantasy numbers). Full analysis and caveats: bench.md Gist by louspringer.
| Band | Output tokens/s | Interpretation |
|---|---|---|
| Poor | < 10 | Underpowered GPU, bad quantization, CPU offload, or VRAM pressure |
| Good | 10–20 | Most self-hosted 70B–120B setups; our 17.3 t/s sits here |
| Strong | 20–35 | High-end GPUs (A100/H100), efficient quantization, clean pipeline |
| Excellent | 35–60+ | Multi-GPU sharding, tensor parallelism, kernel-level optimizations |
| Vendor tier | 100+ | Cloud APIs, batching, speculative decoding — not apples-to-apples |
Bottom line: ~17 output tokens/s = healthy, no obvious misconfiguration; not elite, not broken.
- TTFT — Responsiveness (e.g. 0.5 s vs 8 s) matters for UX; measure with
--ttft. - Concurrency — Throughput can drop under load (e.g. 17 t/s @ 1 user → 9 t/s @ 2 users); use
--concurrent 2or4to probe. - Variance — Use
--runs 5(and optional--warmup) for mean ± std.
Benchmarks and Cursor/Goose use the OpenAI-compatible HTTP API (POST /v1/chat/completions) directly. We do not use LangChain for this:
- Benchmark: Needs precise timing (TTFT, wall time); direct HTTP gives full control and no extra framework overhead.
- Cursor/Goose: Use the configured provider (base URL + model id); no app-level LangChain in this repo.
If you add an application that chains prompts or tools, LangChain (or LangGraph) can be useful there; for measuring and configuring the 120B endpoint, direct HTTP is the right choice.
See: SMOKE_TEST_120B_2026-03-13.md, 120B_SERVE_RUNBOOK.md, GOOSE_LLM_GX10_ACCESS.md.