# Qwen3.8-27B Escha W2 on an RTX 4080 Super 16 GB
## 64k + MTP experiments, 112 Ki context, WSL2, SGLang, FP8 KV, and what actually worked

**Test date:** 2026-09-08  
**Model:** `EschaLabs/Qwen3.8-27B-Escha-W2`  
**Runtime:** `EschaLabs/escha-runtime-qwen3dense` / Escha's SGLang fork  
**GPU:** NVIDIA GeForce RTX 4080 Super 16 GB  
**CPU:** AMD Ryzen 7 7800X3D  
**RAM:** 32 GB  
**Host:** Windows 11  
**Inference environment:** WSL2 / Ubuntu  
**Primary goal:** Run a dense 27B model completely on a 16 GB GPU with at least 64k real context, then determine whether native MTP speculative decoding or more context is the better use of the remaining VRAM.

---

# TL;DR

The best result for my use case was **not** maximum MTP.

On an RTX 4080 Super 16 GB, I ended up with two useful configurations:

### Recommended everyday / long-context mode

- Qwen3.8-27B Escha W2
- **114,688 token real KV pool**
- No MTP
- FP8 E4M3 KV cache
- BF16 Mamba SSM state
- CUDA graphs enabled
- Fully GPU resident
- About **58.5 tok/s** over a 2,048-token generation at a short occupied context
- `nvidia-smi`: about **15,578 MiB / 16,376 MiB**

This gives **112 Ki tokens** of actually allocated context capacity on a 16 GB card.

### Speed mode

- Qwen3.8-27B Escha W2
- **65,536 token real KV pool**
- Native MTP enabled
- `TOPK=1`
- `STEPS=3`
- `DRAFT_TOKENS=4`
- BF16 Mamba SSM state
- FP8 E4M3 KV
- CUDA graphs enabled
- About **67.7 tok/s** averaged over a forced 2,048-token generation
- Roughly **+14.9%** versus the equivalent no-MTP BF16 64k baseline

For coding agents and long-running conversations, I prefer:

> **114,688 context at ~58–59 tok/s**

over:

> **65,536 context at ~68 tok/s**

The former gives **75% more maximum context** for only about a 15% decode-speed sacrifice.

---

# 1. Why I tested this

I have a 16 GB RTX 4080 Super and wanted a local model suitable for serious coding/agent use.

My hard requirement was:

> **At least 64k real context.**

I was not interested in a configuration that merely sets:

```text
context_len=65536

while only allocating enough KV cache for 30k or 40k tokens.

The actual requirement was:

max_total_num_tokens >= 65536

That distinction turned out to be extremely important.

I also wanted to investigate Escha's newly added native Qwen3.8 MTP speculative decoding, because Escha publishes large single-user decode gains on 16 GB cards.

The question was:

On a 16 GB 4080 Super, is VRAM better spent on speculative decoding, or on more context?

2. Relevant Escha/SGLang background

The Escha W2 model is a very low-bit quantization of Qwen3.8-27B designed to make a dense 27B model viable on consumer GPUs.

The current Escha model card reports, under its own benchmark configuration:

GPU	No speculation	Native MTP
RTX 4090	67.3 tok/s	129.3 tok/s
RTX 5080 16 GB	61.7 tok/s	109.2 tok/s
RTX 4080 16 GB	52.8 tok/s	93.4 tok/s

Important: those numbers should not be assumed to represent full-64k-context configurations.

Escha's MTP implementation uses Qwen's own pretrained multi-token-prediction head.

It is not a separately trained small draft model.

The MTP directory is approximately:

0.849 GB

Escha documents MTP as a single-user latency optimization, not a concurrency-throughput optimization.

Escha also notes that speculative-decoding gain falls as input context becomes longer.

That becomes important later.

3. Software environment

The working environment ended up as:

Python:        3.12
PyTorch:       2.9.x + cu128
CUDA toolkit:  12.8 inside WSL
Transformers:  5.16.1
Runtime:       Escha SGLang fork
Model:         Qwen3.8-27B-Escha-W2

Windows NVIDIA side during testing:

Windows NVIDIA driver: 616.64
CUDA UMD:              13.4

WSL nvidia-smi showed:

NVIDIA-SMI: 615.65.07
KMD Version: 616.64
CUDA UMD: 13.4

4. Installation issues encountered

Several dependencies were necessary before the runtime worked correctly.

libnuma

The first failure was:

ImportError: libnuma.so.1

Fixed with:

sudo apt install libnuma1

Compiler / Python headers

The runtime also required a compiler and Python development headers:

sudo apt install build-essential python3.12-dev

nvcc / CUDA toolkit

FlashInfer JIT compilation later failed because nvcc was unavailable:

Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist

Installing the CUDA toolkit 12.8 inside WSL fixed this.

Important:

I installed the CUDA toolkit, not a separate Linux NVIDIA driver.

WSL uses the Windows GPU driver.

The working environment variables were:

export CUDA_HOME=/usr/local/cuda-12.8
export PATH="$CUDA_HOME/bin:$PATH"

Useful preflight checks:

python -c "import sgl_kernel; print('sgl_kernel OK')"

and:

python -c "import torch, escha, sglang; print(torch.cuda.is_available(), hasattr(torch.ops.escha, 'escham_decode_gemv'), bool(sglang.__version__))"

Do not blindly follow the generic runtime warning suggesting Transformers 4.57.1.

Our working Escha environment used:

Transformers 5.16.1

and ran successfully.

5. One of the biggest wins: freeing the 4080 from Windows display duty

Initially the RTX 4080 Super was also driving the Windows desktop.

Idle VRAM consumption was roughly:

~1.8–2.3 GB

That is devastating when trying to squeeze a 27B model, KV cache, recurrent state, CUDA graphs, and MTP into 16 GB.

The Ryzen 7 7800X3D has integrated graphics, so I enabled the iGPU and moved the Windows desktop to the motherboard HDMI output.

After rebooting and moving Windows to the iGPU:

RTX 4080 Super idle VRAM:
0 MiB / 16376 MiB

with:

No running processes found

This was one of the most important changes in the entire experiment.

It effectively gave the inference runtime around another 2 GB of usable GPU memory.

I strongly recommend this over scripts that kill:

explorer.exe
dwm.exe
UxSms

to claw back VRAM.

Running the desktop from the CPU iGPU is much cleaner and avoids Windows display/HDR/desktop instability.

6. Understanding `context_len` versus real context

This caused confusion initially.

SGLang may report:

context_len=65536

while its allocated token pool is:

max_total_num_tokens=31998

That is not a real 64k configuration.

Think of:

context_len

as the configured model ceiling.

The actual backed capacity is:

max_total_num_tokens

For my purposes, a configuration only counted as "64k" when:

max_total_num_tokens >= 65536

I also used:

TRUNCATE=0

because I did not want the server silently trimming oversized prompts.

7. FP8 KV cache memory cost

We used:

--kv-cache-dtype fp8_e4m3

At 65,536 tokens, SGLang reported approximately:

K size: 1.00 GB
V size: 1.00 GB

So the effective target KV cost was roughly:

2 GiB / 65536
≈ 32 KiB per token

This relationship held very consistently.

Examples:

65536 tokens:
K = 1.00 GB
V = 1.00 GB

106496 tokens:
K = 1.63 GB
V = 1.63 GB

110592 tokens:
K = 1.69 GB
V = 1.69 GB

114688 tokens:
K = 1.75 GB
V = 1.75 GB

This made memory planning surprisingly predictable.

8. First working 64k no-MTP configuration

Before turning on speculation, we first proved that the model could genuinely run 64k.

A successful early configuration was:

MODEL=./Qwen3.8-27B-Escha-W2 \
MEM=0.945 \
CTXLEN=65536 \
MAXREQ=1 \
MAXMAMBA=1 \
CUDA_GRAPH_BS=1 \
CHUNK=2048 \
INT8=on \
TRUNCATE=0 \
bash runtime/sglang/serve.sh \
--kv-cache-dtype fp8_e4m3

This produced:

max_total_num_tokens=66235
context_len=65536

so real 64k was proven.

A 512-token benchmark settled around:

~57.3 tok/s

Measured steady decode samples:

Mean:

57.30 tok/s

This became our original no-spec baseline.

9. Initial MTP attempt: why it failed

The first MTP2 attempt used:

DRAFT_TOKENS=2

but allowed SGLang to greedily consume spare memory for KV.

It allocated:

KV Cache #tokens: 77076

which was far more than the 65,536 tokens I actually needed.

The target worker completed graph capture, but then SGLang tried to instantiate the MTP draft worker.

At that point only:

~0.14 GB

remained in its budget.

It died while loading the draft model.

This taught us an important lesson:

When VRAM is very tight, do not let spare memory become unnecessary KV cache before loading MTP.

10. The importance of `--max-total-tokens`

SGLang exposes:

--max-total-tokens

which directly caps the token memory pool.

Instead of asking for:

at least 64k

and allowing SGLang to greedily allocate 70k, 80k, etc., we could explicitly say:

--max-total-tokens 65536

This became essential for fitting speculation.

Important detail:

If the requested value is greater than the amount SGLang can actually profile, SGLang does not magically create memory.

It prints something such as:

max_total_tokens=65536 is larger than the profiled value 60291.
Use the profiled value instead.

So the final:

max_total_num_tokens

still needs to be checked.

11. MTP1: technically works, practically terrible

We tested:

DRAFT_TOKENS=1

At:

MEM=0.91

it achieved:

max_total_num_tokens=69687

so it was comfortably true-64k.

But the benchmark was awful:

accept len: 1.00
accept rate: 1.00

and sustained throughput was only around:

~27.7 tok/s

versus roughly:

~57–59 tok/s

without MTP.

The reason is obvious in hindsight:

If the speculative system only advances one accepted token per verification cycle, it provides no speculative advantage but still pays all the draft-model overhead.

MTP1 is therefore pointless for this model/runtime.

12. MTP2: memory gymnastics and the compiler-cache trick

We next moved to:

TOPK=1
STEPS=1
DRAFT_TOKENS=2

Using TOPK=1 was important.

A larger branching factor costs additional memory and compute.

With top-k 1, a clean linear speculative chain made much more sense on a 16 GB GPU.

At:

MEM=0.925
--max-total-tokens 65536

the target model and 65,536 KV tokens fit.

However, startup initially died while trying to capture the MTP draft CUDA graph.

Later we got the server to start, but the first real request failed inside TorchInductor:

select_top_k_tokens()
→ torch.compile()
→ TorchInductor autotuning
→ benchmark_gpu()
→ temporary L2-cache buffer allocation
→ CUDA driver error: device not ready

The actual model fit.

The problem was a one-time compilation/autotuning workspace allocation.

13. Persistent TorchInductor cache workaround

The clean workaround was to precompile / autotune the speculative path at a smaller KV size where there was temporary VRAM available.

First:

mkdir -p ~/llm/escha-qwen38/torch-cache
export TORCHINDUCTOR_CACHE_DIR=~/llm/escha-qwen38/torch-cache

Then warm the same speculative shape with a smaller token pool.

After the compiled/autotuned kernels were saved, restart at 65,536 with the same:

TORCHINDUCTOR_CACHE_DIR

The tight 64k configuration could then reuse the cached result instead of performing the temporary high-memory autotune again.

This technique was later necessary when experimenting with deeper MTP configurations as well.

14. Working MTP2 configuration

The working MTP2 shape was:

TOPK=1
STEPS=1
DRAFT_TOKENS=2

and the final 64k command was:

export TORCHINDUCTOR_CACHE_DIR=~/llm/escha-qwen38/torch-cache

MODEL=./Qwen3.8-27B-Escha-W2 \
SPEC=1 \
TOPK=1 \
STEPS=1 \
DRAFT_TOKENS=2 \
MEM=0.925 \
CTXLEN=65536 \
MAXREQ=1 \
MAXMAMBA=1 \
CUDA_GRAPH_BS=1 \
CHUNK=2048 \
INT8=on \
TRUNCATE=0 \
bash runtime/sglang/serve.sh \
--kv-cache-dtype fp8_e4m3 \
--max-total-tokens 65536

It worked.

Typical acceptance:

accept len: ~1.68–1.88
accept rate: ~84–94%

So the MTP predictor itself was very good.

But throughput was disappointing.

Steady samples:

Mean:

56.69 tok/s

That was actually slightly slower than no speculation.

Conclusion:

MTP2 predicts well, but two-token speculation is not enough to amortize the draft overhead.

15. The major breakthrough: BF16 Mamba SSM state

MTP3 initially would not fit properly at full 64k.

The runtime showed:

ssm_state size:                  0.28 GB
intermediate_ssm_state_cache:    0.84 GB

That is:

1.12 GB

of recurrent state alone.

SGLang supports:

--mamba-ssm-dtype bfloat16

instead of the default FP32 state.

With BF16, the exact same MTP3 state became:

ssm_state:                  0.14 GB
intermediate_ssm_state:     0.42 GB

Total:

0.56 GB

That reclaimed approximately:

0.56 GB

on a 16 GB card.

This was enormous.

It is what turned MTP3 from essentially impossible at real 64k into a working configuration.

16. BF16 SSM is a precision tradeoff

This is important.

The model/runtime default uses FP32 for the Mamba SSM temporal state.

Changing to:

--mamba-ssm-dtype bfloat16

halves its memory footprint, but it is not mathematically identical to FP32.

BF16 retains FP32-like exponent range but has less mantissa precision.

We did not perform a comprehensive long-context accuracy benchmark comparing FP32 SSM against BF16 SSM.

Therefore:

BF16 SSM should be considered a tested memory/performance optimization, not something we proved to be quality-neutral under every workload.

It worked correctly in all of our functional tests.

17. MTP3 at true 64k

The working shape:

TOPK=1
STEPS=2
DRAFT_TOKENS=3

Final configuration:

export TORCHINDUCTOR_CACHE_DIR=~/llm/escha-qwen38/torch-cache

MODEL=./Qwen3.8-27B-Escha-W2 \
SPEC=1 \
TOPK=1 \
STEPS=2 \
DRAFT_TOKENS=3 \
MEM=0.925 \
CTXLEN=65536 \
MAXREQ=1 \
MAXMAMBA=1 \
CUDA_GRAPH_BS=1 \
CHUNK=2048 \
INT8=on \
TRUNCATE=0 \
bash runtime/sglang/serve.sh \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--max-total-tokens 65536

Successful startup:

KV Cache #tokens: 65536
max_total_num_tokens=65536
context_len=65536
target CUDA graph: success
draft CUDA graph: success
first generation: success

MTP3 benchmark:

60.45 tok/s   accept len 2.27
62.83 tok/s   accept len 2.30
70.84 tok/s   accept len 2.58
73.23 tok/s   accept len 2.67

Average:

66.84 tok/s

Acceptance rate:

~76–89%

This was the first speculative configuration that clearly beat no speculation.

18. Proper control: no-MTP + BF16 SSM

Because MTP3 introduced BF16 SSM, we needed a proper control experiment.

Otherwise we could not know whether the speed increase came from:

MTP

or:

BF16 SSM

The control command was:

MODEL=./Qwen3.8-27B-Escha-W2 \
MEM=0.945 \
CTXLEN=65536 \
MAXREQ=1 \
MAXMAMBA=1 \
CUDA_GRAPH_BS=1 \
CHUNK=2048 \
INT8=on \
TRUNCATE=0 \
bash runtime/sglang/serve.sh \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--max-total-tokens 65536

Steady decode:

Mean:

58.96 tok/s

The variance was tiny.

This proved:

BF16 SSM itself only gave a small performance improvement.

MTP3 was genuinely responsible for most of the ~67 tok/s result.

MTP3 improvement versus this proper control:

66.84 / 58.96 - 1
≈ +13.4%

19. MTP4 at true 64k

TOPK=1
STEPS=3
DRAFT_TOKENS=4

BF16 SSM state:

ssm_state:                  0.14 GB
intermediate_ssm_state:     0.56 GB

It fit.

Final command:

export TORCHINDUCTOR_CACHE_DIR=~/llm/escha-qwen38/torch-cache

MODEL=./Qwen3.8-27B-Escha-W2 \
SPEC=1 \
TOPK=1 \
STEPS=3 \
DRAFT_TOKENS=4 \
MEM=0.925 \
CTXLEN=65536 \
MAXREQ=1 \
MAXMAMBA=1 \
CUDA_GRAPH_BS=1 \
CHUNK=2048 \
INT8=on \
TRUNCATE=0 \
bash runtime/sglang/serve.sh \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--max-total-tokens 65536

Success:

KV Cache #tokens: 65536
max_total_num_tokens=65536
context_len=65536
target CUDA graph: success
draft CUDA graph: success
draft extend graph: success
server generation: success

20. Why short MTP benchmarks were misleading

Initial 512-token MTP4 runs looked very fast.

One run averaged about:

~69 tok/s

Another short run showed:

Average:

~71.9 tok/s

But MTP performance changes substantially depending on how predictable the generated text is.

Therefore, a 512-token test was too short.

We standardized on a forced:

2048 output tokens

with:

"ignore_eos": true

to obtain more decode samples.

21. Standard benchmark command

This became our useful repeatable test:

curl -s http://127.0.0.1:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "escha-qwen38-27b-w2",
    "messages": [
      {
        "role": "user",
        "content": "Explain in detail how a CPU cache hierarchy works and why L1, L2 and L3 caches have different sizes and latencies."
      }
    ],
    "max_tokens": 2048,
    "ignore_eos": true,
    "temperature": 0.6
  }' > /dev/null

This deliberately throws away the generated text.

The point was to observe the server-side decode logs.

22. MTP4 2048-token result

Ignoring the first startup/measurement sample, the steady decode samples were:

Mean:

67.71 tok/s

Minimum:

54.00 tok/s

Maximum:

81.36 tok/s

This variability correlated strongly with speculative acceptance.

Examples:

accept len ~3.15
accept rate ~0.79
→ ~81.36 tok/s

versus:

accept len ~2.15
accept rate ~0.54
→ ~54 tok/s

So MTP4 is very workload-dependent.

Overall speedup against the proper no-MTP BF16 baseline:

67.71 / 58.96 - 1
≈ +14.9%

23. MTP result summary

Measured results:

Configuration	Real context	Avg decode	Approx. result
No MTP, FP32 SSM	>=65,536	57.30 tok/s	Original baseline
No MTP, BF16 SSM	65,536	58.96 tok/s	Best simple 64k baseline
MTP1	>=65,536	~27.7 tok/s	Terrible
MTP2, TOPK1/STEPS1	65,536	56.69 tok/s	Slight regression
MTP3, TOPK1/STEPS2, BF16	65,536	66.84 tok/s*	Useful
MTP4, TOPK1/STEPS3, BF16	65,536	67.71 tok/s	Fastest properly long-tested

* MTP3 was measured using a shorter 512-token test, so MTP3 versus MTP4 is not perfectly apples-to-apples.

The important conclusion was nevertheless clear:

Very shallow speculation does not pay.

and:

At least ~3–4 draft tokens were required before MTP became meaningfully faster on this setup.

24. Why we stopped increasing MTP depth

We considered going to MTP5, MTP6, or MTP7.

We did not conclusively prove those depths impossible.

That distinction matters.

Escha actually recommends trying:

DRAFT_TOKENS=6

first on a 16 GB GPU under its own intended configuration.

However, our requirement was unusual:

Maintain a real 65,536-token pool at the same time.

MTP recurrent-state memory increased predictably with draft depth.

With BF16 SSM:

MTP3 intermediate SSM: ~0.42 GB
MTP4 intermediate SSM: ~0.56 GB

so each additional depth cost roughly another:

~0.14 GB

of intermediate SSM state, plus associated draft/graph/workspace costs.

More importantly, MTP4 only gave us:

~14.9%

average improvement over no-MTP BF16.

At that point the more interesting question became:

Instead of spending more VRAM to chase another few speculative tokens, how much context can the card actually hold?

That turned out to be a much better direction.

25. Long-context exploration

We turned MTP off and retained:

FP8 E4M3 KV
BF16 SSM
MAXREQ=1
MAXMAMBA=1
CUDA graph batch size 1

Then we started increasing actual KV capacity.

26. 104 Ki attempt

104 Ki tokens means:

106496 tokens

First attempt:

MEM=0.96

Result:

requested: 106496
profiled:  105366

We were short by only:

1130 tokens

or roughly:

35 MiB

of target FP8 KV.

27. 104 Ki success

Raising to:

MEM=0.965

worked.

Command:

MODEL=./Qwen3.8-27B-Escha-W2 \
MEM=0.965 \
CTXLEN=106496 \
MAXREQ=1 \
MAXMAMBA=1 \
CUDA_GRAPH_BS=1 \
CHUNK=2048 \
INT8=on \
TRUNCATE=0 \
bash runtime/sglang/serve.sh \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--max-total-tokens 106496

Result:

KV Cache #tokens:       106496
K size:                 1.63 GB
V size:                 1.63 GB
max_total_num_tokens:   106496
context_len:            106496
CUDA graph:             success
first generation:       success

nvidia-smi:

15338 MiB / 16376 MiB

28. 108 Ki / 110,592-token success

110592 tokens

Command:

MODEL=./Qwen3.8-27B-Escha-W2 \
MEM=0.98 \
CTXLEN=110592 \
MAXREQ=1 \
MAXMAMBA=1 \
CUDA_GRAPH_BS=1 \
CHUNK=2048 \
INT8=on \
TRUNCATE=0 \
bash runtime/sglang/serve.sh \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--max-total-tokens 110592

Result:

KV Cache #tokens:       110592
K size:                 1.69 GB
V size:                 1.69 GB
max_total_num_tokens:   110592
context_len:            110592
CUDA graph:             success
first generation:       success

nvidia-smi:

15458 MiB / 16376 MiB

29. Final ceiling test: 112 Ki / 114,688 tokens

We decided on one final push:

114688 tokens

with:

MEM=0.985

Command:

MODEL=./Qwen3.8-27B-Escha-W2 \
MEM=0.985 \
CTXLEN=114688 \
MAXREQ=1 \
MAXMAMBA=1 \
CUDA_GRAPH_BS=1 \
CHUNK=2048 \
INT8=on \
TRUNCATE=0 \
bash runtime/sglang/serve.sh \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--max-total-tokens 114688

And it worked.

Startup:

Mamba ssm_state size: 0.14 GB

KV Cache #tokens: 114688
K size: 1.75 GB
V size: 1.75 GB

max_total_num_tokens=114688
context_len=114688

CUDA graph capture: success
server startup: success
first generation: success

nvidia-smi:

15578 MiB / 16376 MiB

So Windows/WDDM still reported around:

798 MiB

of physical VRAM unused.

SGLang itself reported:

available_gpu_mem=0.00 GB

after its static allocations and graph capture.

These are not contradictory.

available_gpu_mem is SGLang's internal/static budgeting result.

It is not the same thing as:

physical VRAM not currently resident according to nvidia-smi

30. 112 Ki decode benchmark

We then ran the same forced 2,048-token generation with the 114,688-token configuration.

Steady decode samples ranged:

57.91 → 59.29 tok/s

Across 50 steady logged samples:

mean ≈ 58.49 tok/s

This is remarkably close to the 64k no-MTP BF16 result:

64k no-MTP BF16:
58.96 tok/s

112 Ki configured no-MTP BF16:
58.49 tok/s

Difference:

~0.8%

So merely allocating a much larger KV pool did not meaningfully hurt short-context decode speed.

31. Important caveat about the 112 Ki benchmark

This needs to be stated clearly.

The 2,048-token test proved:

The 114,688-token KV pool is genuinely allocated.
CUDA graphs work with it.
The server performs real inference.
Short-context decode remains around 58–59 tok/s.

It did not prove:

58–59 tok/s while 110k tokens are actually occupied

During the test, #full token only reached roughly:

As the actual occupied attention context approaches 100k+, decode performance can change because attention must access much more KV.

Therefore:

114,688 is a proven capacity result, not a benchmark of decode speed at 114k occupied context.

A true 100k-context performance benchmark would require a very large prompt and should be measured separately.

32. Final recommended configurations

A. Recommended default: 112 Ki long-context mode

This is what I intend to use most of the time.

MODEL=./Qwen3.8-27B-Escha-W2 \
MEM=0.985 \
CTXLEN=114688 \
MAXREQ=1 \
MAXMAMBA=1 \
CUDA_GRAPH_BS=1 \
CHUNK=2048 \
INT8=on \
TRUNCATE=0 \
bash runtime/sglang/serve.sh \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--max-total-tokens 114688

Properties:

Model:             Qwen3.8-27B Escha W2
Real context:      114688 tokens / 112 Ki
Speculation:       off
KV:                FP8 E4M3
Mamba SSM:         BF16
CUDA graphs:       enabled
Single-user:       yes
VRAM observed:     ~15578 / 16376 MiB
Short-context
decode observed:   ~58.5 tok/s over 2048-token test

B. Speed preset: 64k + MTP4

Use this when 64k is sufficient and maximum interactive decode speed matters more than additional context.

Assumes the required TorchInductor speculative kernels have already been warmed/cached.

export TORCHINDUCTOR_CACHE_DIR=~/llm/escha-qwen38/torch-cache

MODEL=./Qwen3.8-27B-Escha-W2 \
SPEC=1 \
TOPK=1 \
STEPS=3 \
DRAFT_TOKENS=4 \
MEM=0.925 \
CTXLEN=65536 \
MAXREQ=1 \
MAXMAMBA=1 \
CUDA_GRAPH_BS=1 \
CHUNK=2048 \
INT8=on \
TRUNCATE=0 \
bash runtime/sglang/serve.sh \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--max-total-tokens 65536

Properties:

Real context:     65536
MTP:              enabled
TOPK:             1
STEPS:            3
DRAFT_TOKENS:     4
SSM:              BF16
KV:               FP8 E4M3
Average decode:   ~67.7 tok/s over forced 2048-token test
Speedup:          ~14.9% vs equivalent no-MTP BF16 baseline

33. How to warm a tight MTP configuration

If a speculative shape dies because TorchInductor tries to autotune while VRAM is full, use a smaller temporary KV pool.

Example MTP4 warm-up:

mkdir -p ~/llm/escha-qwen38/torch-cache

export TORCHINDUCTOR_CACHE_DIR=~/llm/escha-qwen38/torch-cache

MODEL=./Qwen3.8-27B-Escha-W2 \
SPEC=1 \
TOPK=1 \
STEPS=3 \
DRAFT_TOKENS=4 \
MEM=0.925 \
CTXLEN=65536 \
MAXREQ=1 \
MAXMAMBA=1 \
CUDA_GRAPH_BS=1 \
CHUNK=2048 \
INT8=on \
TRUNCATE=0 \
bash runtime/sglang/serve.sh \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--max-total-tokens 49152

Run at least one actual generation so the relevant compiled/autotuned path is exercised.

Then stop the server and restart at:

--max-total-tokens 65536

while preserving:

export TORCHINDUCTOR_CACHE_DIR=~/llm/escha-qwen38/torch-cache

This was the difference between several tight MTP configurations crashing and working.

34. Failure modes we encountered and what they meant

`libnuma.so.1` missing

Install:

sudo apt install libnuma1

`gcc` / Python headers missing

Install:

sudo apt install build-essential python3.12-dev

`nvcc` missing

FlashInfer needs to JIT kernels.

Install an appropriate CUDA toolkit in WSL and set:

export CUDA_HOME=/usr/local/cuda-12.8
export PATH="$CUDA_HOME/bin:$PATH"

Do not install another NVIDIA display driver inside WSL.

MTP draft fails while loading weights

If the target model has already consumed almost the entire memory pool, the MTP head has nowhere to load.

Solutions include:

cap KV with --max-total-tokens
reduce draft depth
reduce TOPK
lower recurrent-state memory
reduce context

CUDA graph fails during MTP draft capture

If failure occurs after draft weights load, there may be insufficient temporary workspace for FlashInfer/CUDA graph capture.

Reducing the draft tree or freeing recurrent-state memory can help.

First request crashes in TorchInductor autotuning

We saw:

torch.empty(self.L2_cache_size // 4, ...)
RuntimeError: CUDA driver error: device not ready

This happened because the model technically fit, but TorchInductor wanted temporary benchmarking/autotune memory.

Solution:

pre-warm the exact speculative shape at a smaller KV size and preserve TORCHINDUCTOR_CACHE_DIR.

`max_total_tokens` larger than profiled value

Example:

max_total_tokens=106496 is larger than the profiled value 105366.
Use the profiled value instead.

This means the requested context does not fully fit under the current static-memory budget.

The final authority is:

max_total_num_tokens

Do not call a configuration "106k context" if that value is actually:

35. Do not confuse configured context and usable KV

For reproducible results, always record all three:

context_len
KV Cache #tokens
max_total_num_tokens

My success requirement was:

KV Cache #tokens >= desired context
max_total_num_tokens >= desired context

plus:

CUDA graphs successfully captured
server started
real generation succeeded

Anything less was considered an incomplete result.

36. The FP8 KV warning

Every FP8 run emitted:

Using FP8 KV cache but no scaling factors provided.
Defaulting to scaling factors of 1.0.
This may lead to less accurate results!

We did not provide separate calibrated FP8 KV scaling factors.

Therefore the long-context results should be interpreted as:

successful capacity/performance results using SGLang's FP8 KV behavior with scaling factors defaulting to 1.0.

We did not perform a dedicated quality study comparing this FP8 KV configuration against BF16 KV.

That would be a separate experiment.

37. What we learned about MTP

The most interesting lesson was that:

High acceptance rate does not automatically mean speculative decoding is faster.

MTP2 had excellent acceptance:

~84–94%

yet barely matched or slightly lost to normal decoding.

Why?

Because the draft model itself has a cost.

The number of accepted tokens per verification cycle has to become high enough to amortize that cost.

Observed behavior:

MTP1:
accept len ~1.0
→ catastrophic slowdown

MTP2:
accept len ~1.7–1.9
→ roughly break-even / slightly slower

MTP3:
accept len ~2.3–2.7
→ clear speedup

MTP4:
accept len often ~2–3+
→ ~15% average speedup, but variable

This is why simply enabling MTP is not enough.

The shape matters.

38. Why MTP4 varied from ~54 to ~81 tok/s

Speculative decoding depends on how predictable the current generated sequence is.

During the 2,048-token MTP4 run:

accept len ~3.15
accept rate ~79%
→ ~81 tok/s

while:

accept len ~1.98–2.15
accept rate ~49–54%
→ ~54 tok/s

This is expected speculative-decoding behavior.

A short benchmark can therefore substantially overestimate or underestimate MTP performance depending on what the model happens to generate.

Longer fixed-output tests are much better.

39. Why more context won for my use case

The final comparison is approximately:

64k + MTP4:
~67.7 tok/s

112 Ki no-MTP:
~58.5 tok/s at short occupied context

So MTP4 gives approximately:

+15%

decode speed.

But 112 Ki gives:

114688 / 65536 = 1.75

or:

+75%

maximum context capacity.

For:

coding agents
large repositories
long conversations
tool traces
large prompts
retrieval-heavy workloads

I consider the extra context much more valuable.

Therefore:

My default configuration is 114,688 tokens without MTP.

MTP4 remains a useful speed preset when 64k is sufficient.

40. Things we did NOT prove

To avoid overclaiming:

We did not prove 114,688 is the absolute physical maximum

We deliberately stopped at:

MEM=0.985
114688 tokens

because it was already extremely aggressive.

There may be ways to squeeze slightly more context by changing other memory consumers.

We chose stability and usefulness over finding the last theoretical token.

We did not prove MTP5/MTP6/MTP7 are impossible

We stopped after MTP4 because the tradeoff stopped being compelling for my workload.

Deeper speculation might work if:

context is reduced
other state is compressed
graph/workspace use changes
runtime implementation improves

Escha itself suggests DRAFT_TOKENS=6 as a starting point on 16 GB cards under its intended benchmark setup.

Our requirement was different:

keep full real 64k simultaneously.

We did not benchmark decode at 100k+ occupied context

The 112 Ki test allocated the full pool, but the benchmark itself only occupied about 2k tokens.

A separate long-prompt benchmark is needed to measure actual decode throughput near 100k context occupancy.

We did not establish BF16 SSM quality equivalence to FP32

BF16 SSM worked and saved a huge amount of memory.

But a serious accuracy/long-context comparison remains worth doing.

We did not establish FP8 KV quality equivalence to BF16 KV

The runtime warned that no FP8 KV scaling factors were supplied.

Our findings here concern fit, stability and performance, not a full numerical-accuracy evaluation.

41. Final measured scoreboard

Mode	Actual KV capacity	SSM	MTP	Measured decode
Baseline	>=65,536	FP32	Off	57.30 tok/s
64k control	65,536	BF16	Off	58.96 tok/s
MTP1	>=65,536	FP32	1 token	~27.7 tok/s
MTP2	65,536	FP32	1-1-2	56.69 tok/s
MTP3	65,536	BF16	1-2-3	66.84 tok/s*
MTP4	65,536	BF16	1-3-4	67.71 tok/s
Long context	106,496	BF16	Off	boot/proven
Long context	110,592	BF16	Off	boot/proven
Final long context	114,688	BF16	Off	58.49 tok/s

* MTP3 used a shorter benchmark than the final standardized MTP4 test.

42. Final VRAM observations

Representative nvidia-smi values:

GPU total:
16376 MiB

After moving Windows to the 7800X3D iGPU:

Idle RTX 4080 Super:
0 MiB

Selected running configurations:

MTP1 / 64k:
~15734 MiB

MTP2 / 64k:
~15812 MiB

104 Ki no-MTP:
~15338 MiB

108 Ki / 110592 no-MTP:
~15458 MiB

112 Ki / 114688 no-MTP:
~15578 MiB

This illustrates that the runtime's:

available_gpu_mem=0.00 GB

does not mean nvidia-smi must show 16,376 MiB used.

SGLang's memory fraction controls its own static allocation policy.

43. Useful mental model for tuning

On this particular model/card/runtime:

KV

Approximate target FP8 KV cost:

~32 KiB/token

Therefore:

4096 more tokens ≈ 128 MiB
8192 more tokens ≈ 256 MiB

This matched our experiments well.

BF16 MTP SSM depth

Each extra draft depth added roughly:

~0.14 GB

of intermediate BF16 SSM state in our setup.

This made it possible to estimate whether a deeper MTP configuration was even plausible before launching it.

44. Recommended workflow for other 16 GB users

If trying to reproduce this:

Make the NVIDIA GPU compute-only if possible.
Run the desktop from an iGPU.
Verify idle NVIDIA VRAM.
Start with no speculation.
Use FP8 KV.
Verify max_total_num_tokens, not just context_len.
Set TRUNCATE=0 while validating context.
Use --max-total-tokens to prevent SGLang greedily spending memory you need elsewhere.
For hybrid Mamba/GDN models, consider BF16 SSM if memory-bound.
If enabling MTP, start with TOPK=1.
Benchmark MTP over long fixed output lengths.
Watch accept len, not just accept rate.
If tight MTP crashes during first compile/autotune, pre-warm at smaller KV and persist the TorchInductor cache.
Do not assume Escha's published MTP speed numbers will reproduce at your maximum context.
Decide whether your workload values context or decode latency more.

45. My final recommendation

For an RTX 4080 Super 16 GB used for coding, agents and long conversations:

Use this most of the time

MODEL=./Qwen3.8-27B-Escha-W2 \
MEM=0.985 \
CTXLEN=114688 \
MAXREQ=1 \
MAXMAMBA=1 \
CUDA_GRAPH_BS=1 \
CHUNK=2048 \
INT8=on \
TRUNCATE=0 \
bash runtime/sglang/serve.sh \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--max-total-tokens 114688

Keep this as a speed preset

export TORCHINDUCTOR_CACHE_DIR=~/llm/escha-qwen38/torch-cache

MODEL=./Qwen3.8-27B-Escha-W2 \
SPEC=1 \
TOPK=1 \
STEPS=3 \
DRAFT_TOKENS=4 \
MEM=0.925 \
CTXLEN=65536 \
MAXREQ=1 \
MAXMAMBA=1 \
CUDA_GRAPH_BS=1 \
CHUNK=2048 \
INT8=on \
TRUNCATE=0 \
bash runtime/sglang/serve.sh \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--max-total-tokens 65536

46. Bottom line

The surprising result of this whole exercise was not the MTP speedup.

It was this:

A dense Qwen3.8-27B model can run with a real 114,688-token KV pool on an RTX 4080 Super 16 GB, entirely GPU resident, with CUDA graphs enabled, while still decoding at roughly 58–59 tok/s at ordinary occupied context lengths.

Native MTP also works at real 64k.

With a carefully memory-constrained TOPK=1, STEPS=3, DRAFT_TOKENS=4 configuration, we measured about:

67.7 tok/s

over a forced 2,048-token test, approximately:

+14.9%

over the equivalent no-MTP BF16 baseline.

But for my workload, the better trade is:

112 Ki context @ ~58–59 tok/s

rather than:

64k context @ ~68 tok/s

The extra context is simply more useful.

References / projects

The relevant upstream projects and documentation are:

EschaLabs/Qwen3.8-27B-Escha-W2 on Hugging Face
EschaLabs/escha-runtime-qwen3dense on Hugging Face
SGLang speculative-decoding documentation
SGLang server-argument documentation
PyTorch TorchInductor / Triton compilation and caching behavior

All performance and memory numbers described as "measured" above came from the actual RTX 4080 Super system described in this document.

They should be treated as one reproducible enthusiast test setup, not universal benchmark claims.

Reproduction note

When sharing results, please include:

GPU model
GPU VRAM
whether the GPU is driving a display
Windows/Linux/WSL
driver version
PyTorch version
Escha runtime version
Transformers version
KV dtype
SSM dtype
MEM
CTXLEN
max_total_num_tokens
MAXREQ
MAXMAMBA
CUDA graph settings
MTP TOPK/STEPS/DRAFT_TOKENS
actual occupied context during benchmark

Without those details, speculative-decoding and long-context numbers are very easy to compare incorrectly.




## Update: 112 Ki boots, but is not a stable everyday ceiling

After publishing the initial 114,688-token result, I tested the configuration with a normal frontend rather than only isolated benchmark requests.

The 112 Ki configuration was:

```bash
MODEL=./Qwen3.8-27B-Escha-W2 \
MEM=0.985 \
CTXLEN=114688 \
MAXREQ=1 \
MAXMAMBA=1 \
CUDA_GRAPH_BS=1 \
CHUNK=2048 \
INT8=on \
TRUNCATE=0 \
bash runtime/sglang/serve.sh \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--max-total-tokens 114688

It successfully:

allocated all 114688 KV tokens
captured the CUDA graph
started the server
completed isolated generations
completed a forced 2048-token benchmark

However, under normal frontend traffic it later crashed during a new prefill/extend operation:

forward_extend
→ chunk_gated_delta_rule
→ solve_tril
→ torch.empty(...)
→ RuntimeError: CUDA driver error: device not ready

The important startup clue was:

available_gpu_mem=0.00 GB

The model and KV cache technically fit, but the configuration left essentially no SGLang runtime headroom for temporary buffers required by GDN/linear-attention prefill kernels.

The frontend also had another request queued:

#queue-req: 1

while one generation was running. MAXREQ=1 limits concurrent running requests, but queued requests can still arrive and subsequently require their own prefill/extend workspace.

Revised conclusion

The 114,688-token result should therefore be described as:

Experimental maximum demonstrated: 114,688 tokens / 112 Ki
Boots, captures CUDA graphs, and handles isolated inference, but is too close to the VRAM limit for reliable general-purpose server use.

It should not be recommended as the everyday configuration.

The practical long-context target is now:

110,592 tokens / 108 Ki

with enough additional runtime margin to be a much better daily-use candidate.

So the revised hierarchy is:

114,688 / 112 Ki
✅ real KV allocation
✅ CUDA graphs
✅ isolated inference
❌ insufficient runtime headroom under normal frontend traffic
→ experimental ceiling only

110,592 / 108 Ki
✅ real KV allocation
✅ CUDA graphs
✅ recommended long-context target

65,536 + MTP4
✅ real 64k
✅ ~67.7 tok/s measured
→ speed preset

This was a useful distinction:

A configuration that successfully loads is not necessarily a configuration with enough transient VRAM headroom to operate reliably as a real server.

Update: 96 Ki / 98,304 tokens is the recommended real-world configuration

Further testing changed the recommendation.

The 114,688-token and 110,592-token configurations could allocate their full KV pools, capture CUDA graphs, start successfully, and complete isolated inference. However, both left effectively zero SGLang runtime headroom and crashed during normal GDN prefill/extend operations under real client traffic.

The configuration that has proven substantially more robust is:

98,304 tokens / 96 Ki context

It leaves approximately 0.22 GB of SGLang runtime headroom after CUDA graph capture, while still providing 50% more context than 64k.

Recommended command

MODEL=./Qwen3.8-27B-Escha-W2 \
MEM=0.98 \
CTXLEN=98304 \
MAXREQ=1 \
MAXMAMBA=1 \
CUDA_GRAPH_BS=1 \
CHUNK=2048 \
INT8=on \
TRUNCATE=0 \
bash runtime/sglang/serve.sh \
--kv-cache-dtype fp8_e4m3 \
--mamba-ssm-dtype bfloat16 \
--max-total-tokens 98304

Startup result:

KV Cache #tokens:       98304
K size:                 1.50 GB
V size:                 1.50 GB
max_total_num_tokens:   98304
context_len:            98304
available_gpu_mem:      0.22 GB
CUDA graph:             successful

Unlike the higher-context experiments, this configuration has also survived real Hermes usage with large prompts and repeated requests.

Observed real-context decode performance:

~20k occupied context:  ~54–55 tok/s
~26–28k context:        ~54.6 tok/s
~51k context:           ~51.3–51.4 tok/s

At approximately 51,000 live tokens:

#full token:       ~51200
full token usage:  ~0.52
gen throughput:    ~51.4 tok/s

This is especially encouraging because it is no longer a synthetic short-context benchmark: roughly half of the 98,304-token pool is actually occupied.

Revised recommendation

The configurations should now be classified as:

114,688 / 112 Ki
Experimental allocation ceiling only.
Boots and performs isolated inference, but insufficient runtime workspace.

110,592 / 108 Ki
Also too aggressive for normal server usage.
Real frontend traffic can exhaust temporary GDN prefill workspace.

98,304 / 96 Ki
RECOMMENDED LONG-CONTEXT CONFIGURATION.
Real KV capacity, CUDA graphs, useful runtime headroom,
and proven under actual Hermes workloads.

65,536 + MTP4
Optional speed configuration when 64k context is sufficient.
~67.7 tok/s measured in our speculative-decoding benchmark.

Bottom line

For this RTX 4080 Super 16 GB setup, 98,304 tokens is currently the configuration to run.

It gives up some theoretical maximum context in exchange for something much more important:

enough transient GPU memory for the model to actually operate reliably under real workloads.

The practical result is a dense Qwen3.8-27B model running entirely on a 16 GB GPU with a real 96 Ki context window, while still delivering roughly 51 tok/s at ~51k occupied context.

pyed/gist:831454fa2171e67ff232c0ac9330f7e6

2. Relevant Escha/SGLang background

3. Software environment

4. Installation issues encountered

libnuma

Compiler / Python headers

nvcc / CUDA toolkit

5. One of the biggest wins: freeing the 4080 from Windows display duty

6. Understanding context_len versus real context

7. FP8 KV cache memory cost

8. First working 64k no-MTP configuration

9. Initial MTP attempt: why it failed

10. The importance of --max-total-tokens

11. MTP1: technically works, practically terrible

12. MTP2: memory gymnastics and the compiler-cache trick

13. Persistent TorchInductor cache workaround

14. Working MTP2 configuration

15. The major breakthrough: BF16 Mamba SSM state

16. BF16 SSM is a precision tradeoff

17. MTP3 at true 64k

18. Proper control: no-MTP + BF16 SSM

19. MTP4 at true 64k

20. Why short MTP benchmarks were misleading

21. Standard benchmark command

22. MTP4 2048-token result

23. MTP result summary

24. Why we stopped increasing MTP depth

25. Long-context exploration

26. 104 Ki attempt

27. 104 Ki success

28. 108 Ki / 110,592-token success

29. Final ceiling test: 112 Ki / 114,688 tokens

30. 112 Ki decode benchmark

31. Important caveat about the 112 Ki benchmark

32. Final recommended configurations

A. Recommended default: 112 Ki long-context mode

B. Speed preset: 64k + MTP4

33. How to warm a tight MTP configuration

34. Failure modes we encountered and what they meant

libnuma.so.1 missing

gcc / Python headers missing

nvcc missing

MTP draft fails while loading weights

CUDA graph fails during MTP draft capture

First request crashes in TorchInductor autotuning

max_total_tokens larger than profiled value

35. Do not confuse configured context and usable KV

36. The FP8 KV warning

37. What we learned about MTP

38. Why MTP4 varied from ~54 to ~81 tok/s

39. Why more context won for my use case

40. Things we did NOT prove

We did not prove 114,688 is the absolute physical maximum

We did not prove MTP5/MTP6/MTP7 are impossible

We did not benchmark decode at 100k+ occupied context

We did not establish BF16 SSM quality equivalence to FP32

We did not establish FP8 KV quality equivalence to BF16 KV

41. Final measured scoreboard

42. Final VRAM observations

43. Useful mental model for tuning

KV

BF16 MTP SSM depth

44. Recommended workflow for other 16 GB users

45. My final recommendation

Use this most of the time

Keep this as a speed preset

46. Bottom line

References / projects

Reproduction note

Revised conclusion

Update: 96 Ki / 98,304 tokens is the recommended real-world configuration

Recommended command

Revised recommendation

Bottom line

6. Understanding `context_len` versus real context

10. The importance of `--max-total-tokens`

`libnuma.so.1` missing

`gcc` / Python headers missing

`nvcc` missing

`max_total_tokens` larger than profiled value