State of the Qwen-ion

turboquant_qwen36_old_gpus.md

STATE OF THE QWEN-ION

a.k.a

Qwen3.6-35B-A3B Q4_K_M on RTX 3080 + Tesla V100 with TurboQuant KV Cache: Practical Reproduction Report

This report documents a reproducible Docker-based setup for running Qwen3.6-35B-A3B Q4_K_M with TheTom's llama-cpp-turboquant fork on a mixed NVIDIA GPU system.

It includes both:

1. the runtime and benchmark results from the tested configuration; and
2. the Docker build and environment setup required to create the working TurboQuant server image.

The goal is to let another user reproduce the same class of setup without relying on host-specific paths, usernames, private network details, or unpublished local state.

Table of contents

• Qwen3.6-35B-A3B Q4_K_M on RTX 3080 + Tesla V100 with TurboQuant KV Cache: Practical Reproduction Report
• De-identification and example convention
• Tested system profile
• Key result
• Why q8_0/turbo3 won
• Repository and Docker image provenance
• Prerequisites
• Build the Docker image
  • Verify that the image contains the TurboQuant cache options
• Optional fallback: minimal manual Dockerfile
• Run the server
• Wait for readiness
• Basic API test
• Benchmark results
• Usage summary
• Reproducibility notes
• Short checklist
• Troubleshooting
  • The Docker image does not include TurboQuant options
  • Docker cannot see the GPUs
  • The server starts but output quality is bad
  • The server fails to start at 128K
  • The GPUs are not balanced well
  • Speculative decoding is tempting but fails validation
• Conclusion

De-identification and example convention

This public version avoids personal details, hostnames, account names, private LAN addresses, and machine-specific filesystem paths.

Example values use generic placeholders such as:

• /path/to/Qwen3.6-35B-A3B-Q4_K_M.gguf
• $WORKDIR
• qwen36-35b-a3b-tq
• thetom-llama-cpp-turboquant:cuda-server
• localhost:18084

Replace these with local values before running commands.

The benchmark results are tied only to the hardware and software profile described below. They are not tied to a particular operator, host, LAN, account, or private dataset.

Tested system profile

Hardware profile:

• GPU 0: NVIDIA GeForce RTX 3080, 10 GB VRAM
• GPU 1: NVIDIA Tesla V100, 16 GB VRAM
• System RAM: approximately 47 GB
• GPU mode: mixed NVIDIA GPUs with Docker GPU passthrough
• Model format: GGUF, Q4_K_M
• Runtime: Docker container built locally from TheTom's llama-cpp-turboquant fork

CUDA architecture mapping for this GPU pair:

GPU	CUDA architecture
Tesla V100	70
GeForce RTX 3080	86

The Docker image build should include both architectures:

--build-arg CUDA_DOCKER_ARCH="70;86"

For different GPUs, replace 70;86 with the architectures for the target hardware.

Key result

The fastest stable configuration was:

• K cache type: -ctk q8_0
• V cache type: -ctv turbo3
• Context: -c 131072 / 128K
• Split mode: --split-mode layer
• Tensor split: --tensor-split 10,16
• Batching: -b 2048 -ub 512
• Flash attention: -fa on
• Speculative decoding: disabled with --spec-type none
• Reasoning mode: disabled with --reasoning off --reasoning-budget 0
• Observed short extraction performance: approximately 104.8 tokens/s
• Long-context validation: PASS on a 130,717-token canary test, with all canary parts found

Other tested configurations, including symmetric TurboQuant cache types, q8_0/turbo4, speculative modes, larger batches, and alternative tensor splits, were slower or failed validation.

Why q8_0/turbo3 won

The winning setup used asymmetric KV cache quantisation:

• K = q8_0
• V = turbo3

This kept key-cache precision high enough for stable long-context recall while compressing the value cache enough to make 128K context practical and fast on the tested hardware.

In testing:

• q8_0/turbo3 was fastest and passed the long-context canary.
• q8_0/turbo4 passed, but was slower.
• turbo3/turbo3 passed at 128K, but was noticeably slower.
• Lower-K and speculative decoding variants were either slower, unstable, or failed the canary.

Practical rule for this hardware class:

Use q8_0 for K.
Use turbo3 for V.
Disable speculative decoding.
Use layer split mode.
Use tensor split 10,16.
Use batch 2048 and ubatch 512.

Repository and Docker image provenance

The tested Docker image was not assumed to be an official prebuilt registry image. It was built locally from TheTom's llama-cpp-turboquant fork.

Repository:

https://github.com/TheTom/llama-cpp-turboquant

Branch used:

feature/turboquant-kv-cache

Local Docker image tag used in this report:

thetom-llama-cpp-turboquant:cuda-server

This tag is a local name. It does not imply that an image with this name exists in a public registry.

Prerequisites

Install or provide:

• Docker
• NVIDIA driver
• NVIDIA Container Toolkit
• Git
• A GGUF model file for Qwen3.6-35B-A3B Q4_K_M
• Enough free disk space for the source tree and Docker build cache

Check that Docker can see the GPUs:

docker run --rm --gpus all nvidia/cuda:12.6.3-base-ubuntu22.04 nvidia-smi

If Docker requires root on the target system, prefix Docker commands with sudo.

Build the Docker image

Use a generic work directory. This avoids embedding private host paths into the public instructions.

WORKDIR="$HOME/turboquant-work"
REPO="$WORKDIR/llama-cpp-turboquant"
IMAGE="thetom-llama-cpp-turboquant:cuda-server"
BRANCH="feature/turboquant-kv-cache"

mkdir -p "$WORKDIR"
cd "$WORKDIR"

Clone or update the repo:

if [ ! -d "$REPO/.git" ]; then
  git clone \
    --branch "$BRANCH" \
    https://github.com/TheTom/llama-cpp-turboquant.git \
    "$REPO"
else
  cd "$REPO"
  git fetch origin
  git checkout "$BRANCH"
  git pull --ff-only
fi

Build the CUDA server image:

cd "$REPO"

DOCKER_BUILDKIT=1 docker build \
  --build-arg CUDA_DOCKER_ARCH="70;86" \
  --target server \
  -f .devops/cuda.Dockerfile \
  -t "$IMAGE" \
  .

This uses BuildKit and builds the repo's CUDA Docker server target.

Verify that the image contains the TurboQuant cache options

After building, confirm that the server exposes the expected cache controls:

docker run --rm "$IMAGE" --help 2>&1 | grep -Ei 'ctk|ctv|cache-type|turbo|kv'

Expected result: the help output should mention -ctk, -ctv, cache types, or TurboQuant-related KV cache options.

If the output does not show those options, the wrong branch or wrong image was built.

Check the branch directly if needed:

cd "$REPO"
git branch --show-current
git log -1 --oneline

Optional fallback: minimal manual Dockerfile

The in-repo Dockerfile is the preferred build path. If the repo layout changes or .devops/cuda.Dockerfile is unavailable, a minimal manual build can be used as a fallback.

Create a fallback Dockerfile in an empty directory:

cat > Dockerfile <<'EOF_DOCKER'
FROM nvidia/cuda:12.6.3-devel-ubuntu22.04

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y --no-install-recommends \
    ca-certificates \
    cmake \
    build-essential \
    git \
    curl \
    python3 \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /src

RUN git clone \
    --branch feature/turboquant-kv-cache \
    --depth=1 \
    https://github.com/TheTom/llama-cpp-turboquant.git

WORKDIR /src/llama-cpp-turboquant

RUN cmake -B build \
    -DGGML_CUDA=ON \
    -DCMAKE_CUDA_ARCHITECTURES="70;86" \
    -DCMAKE_BUILD_TYPE=Release \
    && cmake --build build --config Release -j"$(nproc)" --target llama-server llama-cli llama-bench

RUN cp build/bin/llama-server build/bin/llama-cli build/bin/llama-bench /usr/local/bin/

WORKDIR /models
EXPOSE 8080

ENTRYPOINT ["llama-server"]
EOF_DOCKER

Build it:

DOCKER_BUILDKIT=1 docker build \
  -t "thetom-llama-cpp-turboquant:cuda-server" \
  .

Then run the same help check:

docker run --rm "thetom-llama-cpp-turboquant:cuda-server" --help 2>&1 | grep -Ei 'ctk|ctv|cache-type|turbo|kv'

Use the in-repo Docker build when possible. The fallback exists only to document how the environment can be recreated if the repo Docker target is not available.

Run the server

Set the model path and container parameters:

MODEL_PATH="/path/to/Qwen3.6-35B-A3B-Q4_K_M.gguf"
IMAGE="thetom-llama-cpp-turboquant:cuda-server"
CONTAINER_NAME="qwen36-35b-a3b-tq"
HOST_PORT="18084"

Remove any previous test container with the same name:

docker rm -f "$CONTAINER_NAME" 2>/dev/null || true

This is intentionally scoped to the named test container. Do not remove every Docker container on a shared or production host.

Run the server:

docker run -d \
  --name "$CONTAINER_NAME" \
  --gpus all \
  --cap-add IPC_LOCK \
  --ulimit memlock=-1:-1 \
  -p "$HOST_PORT:8080" \
  --mount type=bind,src="$MODEL_PATH",dst="/models/model.gguf",readonly \
  "$IMAGE" \
  -m "/models/model.gguf" \
  --alias qwen36-35b-a3b \
  -ngl 999 \
  -c 131072 \
  -np 1 \
  -b 2048 \
  -ub 512 \
  -t 12 \
  -tb 12 \
  -fa on \
  --split-mode layer \
  --tensor-split 10,16 \
  --main-gpu 0 \
  --kv-unified \
  --spec-type none \
  --reasoning off \
  --reasoning-budget 0 \
  --mlock \
  -ctk q8_0 \
  -ctv turbo3 \
  --metrics

This launches the server on:

http://localhost:18084

Wait for readiness

Poll /health before sending workloads:

for i in {1..60}; do
  if curl -fsS "http://localhost:18084/health" | grep -q '"status":"ok"'; then
    echo "Server is ready"
    break
  fi
  sleep 2
done

Check the loaded model endpoint:

curl -s "http://localhost:18084/v1/models" | jq .

If jq is not installed:

curl -s "http://localhost:18084/v1/models"

Basic API test

curl -s "http://localhost:18084/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen36-35b-a3b",
    "messages": [
      {
        "role": "user",
        "content": "Reply with one short sentence confirming the server is working."
      }
    ],
    "max_tokens": 64,
    "temperature": 0
  }' | jq .

If jq is not installed, omit the pipe:

curl -s "http://localhost:18084/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen36-35b-a3b",
    "messages": [
      {
        "role": "user",
        "content": "Reply with one short sentence confirming the server is working."
      }
    ],
    "max_tokens": 64,
    "temperature": 0
  }'

Benchmark results

Short extraction testing used a 12-record extraction workload and returned valid JSON. The long validation test used an approximately 130K-token prompt with multiple canary phrases placed throughout the context.

Only configurations that produced valid output and found all canary phrases were kept.

Test	Setup	Token/s, short runs	Mean	Canary result	Notes
Winner	ctk q8_0, ctv turbo3, no speculative decoding, split 10,16, b2048/ub512	104.60 / 104.77 / 104.89	104.75	PASS	Stable and fast at 128K
q8_0/turbo4	same except ctv turbo4	95.6 / 98.1 / 97.7	~97.1	PASS	Stable, but about 7% slower
turbo3/turbo3 at 32K	turbo3/turbo3, -c 32768	100.6 / 101.0 / 101.0	100.9	not tested	Worked at 32K; KV cache much smaller
turbo3/turbo3 at 128K	turbo3/turbo3, -c 131072	92.3 / 92.3 / 92.1	92.2	PASS	Much slower
Other tested variants	lower-K such as q5_1/turbo3, larger batches, split 11,16, n-gram speculative decoding	mixed	mixed	fail or slower	Not better than winner

Usage summary

• Build: build the local CUDA Docker image from TheTom/llama-cpp-turboquant, branch feature/turboquant-kv-cache.
• Image tag: this report uses thetom-llama-cpp-turboquant:cuda-server as a local image tag.
• Production command: use the docker run command above and replace only the model path, image name, container name, and port if needed.
• Health check: ensure /health returns OK before sending workloads.
• Benchmarking: run both a short performance test and a long-context canary test.
• Validation: keep only configurations that produce valid structured output and pass the long-context canary.
• Result: the chosen config achieved approximately 105 tokens/s on the short extraction workload and passed the full long-context canary validation.

Reproducibility notes

For comparable results:

• Use the same model quantisation: Q4_K_M.
• Use the same GPU class and order, or adjust --tensor-split and --main-gpu.
• Keep --split-mode layer.
• Keep --tensor-split 10,16 for a 10 GB plus 16 GB VRAM layout.
• Keep -b 2048 -ub 512 unless deliberately benchmarking alternatives.
• Keep -ctk q8_0 -ctv turbo3 for the winning setup.
• Disable speculative decoding with --spec-type none.
• Disable reasoning with --reasoning off --reasoning-budget 0.
• Check /health before each benchmark.
• Run a long-context canary test, not just a short speed test.

Throughput alone is not enough. A configuration that is fast but loses canaries is not valid for long-context work.

Short checklist

Use this as the starting point:

-ctk q8_0
-ctv turbo3
-c 131072
--split-mode layer
--tensor-split 10,16
-b 2048
-ub 512
-fa on
--spec-type none
--reasoning off
--reasoning-budget 0
--kv-unified

Expected result on the tested hardware profile:

Approximately 104-105 tokens/s on the short extraction workload.
PASS on the approximately 130K-token canary validation.

Troubleshooting

The Docker image does not include TurboQuant options

Confirm the branch:

cd "$REPO"
git branch --show-current
git log -1 --oneline

Confirm the binary help output:

docker run --rm "$IMAGE" --help 2>&1 | grep -Ei 'ctk|ctv|cache-type|turbo|kv'

If the help output does not show the expected options, rebuild from the TurboQuant KV-cache branch.

Docker cannot see the GPUs

Check the NVIDIA container runtime:

docker run --rm --gpus all nvidia/cuda:12.6.3-base-ubuntu22.04 nvidia-smi

If this fails, fix NVIDIA Container Toolkit before debugging the model server.

The server starts but output quality is bad

Try the safer baseline:

-ctk q8_0 -ctv turbo4

If that works but q8_0/turbo3 does not, the model, branch, or GPU layout may behave differently from the tested setup.

The server fails to start at 128K

Reduce context first:

-c 65536

Then test:

-c 32768

If lower context works, the issue is likely VRAM/RAM pressure, GPU split, or KV cache allocation.

The GPUs are not balanced well

Check GPU memory usage:

nvidia-smi

Then adjust:

--tensor-split 10,16

The tested split matched a 10 GB GPU plus a 16 GB GPU. Other hardware should use a split appropriate to available VRAM.

Speculative decoding is tempting but fails validation

The tested speculative decoding modes did not beat the winning non-speculative setup and could fail validation. Keep this disabled unless deliberately testing it:

--spec-type none

Conclusion

For the tested RTX 3080 plus Tesla V100 system, the best validated setup was:

K = q8_0
V = turbo3
Context = 128K
Split = layer mode, tensor split 10,16
Batch = 2048
Ubatch = 512
Speculative decoding = off
Reasoning = off

This combination gave the best observed throughput while preserving long-context recall in the canary validation test.

The missing reproduction detail was the Docker image provenance: build the CUDA server image locally from TheTom's llama-cpp-turboquant fork, branch feature/turboquant-kv-cache, with CUDA architectures 70;86 for the tested V100 plus RTX 3080 hardware.