Local Qwen 3.6 35B-A3B (Q8) Setup with llama.cpp, MTP & Vision

qwen3.6-llamacpp-mtp-vision.md

Local Qwen 3.6 35B-A3B (Q8) Setup with llama.cpp, MTP & Vision

A complete, reproducible guide for running Qwen 3.6 35B-A3B (Q8_K quant) locally with Multi-Token Prediction (MTP) and image/vision support using llama.cpp.

---

📥 1. Download from Hugging Face

# Option A: huggingface-cli (recommended for large models)
huggingface-cli download <YOUR-HF-REPO> \
  --local-dir ./qwen3.6-35b-a3b-q8 \
  --exclude "*.safetensors" "*.py" "*.md" "model*.json"

# Option B: git-lfs (if hosted as a git repo)
git clone https://huggingface.co/<YOUR-HF-REPO> ./qwen3.6-35b-a3b-q8
cd qwen3.6-35b-a3b-q8 && git lfs pull

# Convert to GGUF (if needed)
python3 convert_hf_to_gguf.py . \
  --outfile ./qwen3.6-35b-a3b-q8.gguf \
  --outtype f16 --mtp

# Quantize to Q8 (or Q5_K_M for speed)
llama-quantize ./qwen3.6-35b-a3b-q8.gguf \
  ./qwen3.6-35b-a3b-q8_q8.gguf Q8_K_M

---

🛠️ 2. Build & Install llama.cpp

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
mkdir build && cd build
cmake .. -DBUILD_SHARED_LIBS=OFF \
         -DLLAMA_CURL=ON \
         -DLLAMA_CUDA=ON \
         -DLLAMA_VULKAN=ON \
         -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)

---

🔧 3. Configuration Files

config.json (model root)

{
  "architectures": ["Qwen3ForCausalLM"],
  "model_type": "qwen3",
  "moe_intermediate_size": 128,
  "num_experts_per_tok": 2,
  "num_key_value_heads": 8,
  "hidden_size": 8192,
  "num_hidden_layers": 48,
  "num_attention_heads": 64,
  "max_position_embeddings": 131072,
  "rms_norm_eps": 1e-5,
  "rope_theta": 1000000.0,
  "vocab_size": 152064
}

params.json (generation overrides)

{
  "temperature": 0.7,
  "top_p": 0.9,
  "top_k": 40,
  "repeat_penalty": 1.1,
  "repeat_last_n": 64,
  "mirostat": 2,
  "mirostat_tau": 5.0,
  "mirostat_eta": 0.1
}

---

🔮 4. MTP (Multi-Token Prediction) Setup

MTP requires the model to have MTP heads baked in during conversion. Enable with these flags:

# MTP enables speculative token generation for ~2-3x throughput
# Works best with Q5_K_M or Q8 quantizations

# Enable MTP
--mtp \
--mtp-nb-heads 4 \
--mtp-nb-tokens 16 \
--mtp-temp 0.9

# If you see instability, lower MTP temperature or reduce heads:
# --mtp-nb-heads 2 --mtp-nb-tokens 8 --mtp-temp 0.7

---

👁️ 5. Vision / Image Setup

# 1. Extract mmproj from model (if not already separated)
python3 extract_mmproj.py ./qwen3.6-35b-a3b-q8_q8.gguf ./qwen3.6-35b-a3b-mmproj.gguf

# 2. Run with image input
--mmproj ./qwen3.6-35b-a3b-mmproj.gguf \
--image ./screenshot.png \
--image-format png \
--ctx-size 16384 \
--batch-size 512

# 3. Multi-image / video frame piping
cat frame_1.jpg frame_2.jpg | llama-cli \
  --mmproj ./qwen3.6-35b-a3b-mmproj.gguf \
  --batch-size 1024 \
  --threads $(nproc)

---

🚀 6. Launch Commands

Base Server (Web UI / API)

llama-server \
  --model ./qwen3.6-35b-a3b-q8_q8.gguf \
  --ctx-size 32768 \
  --batch-size 1024 \
  --threads $(nproc) \
  --port 8080 \
  --host 0.0.0.0

MTP Optimized (High Throughput)

llama-server \
  --model ./qwen3.6-35b-a3b-q8_q8.gguf \
  --mtp --mtp-nb-heads 4 --mtp-nb-tokens 16 --mtp-temp 0.9 \
  --ctx-size 16384 \
  --batch-size 512 \
  --threads $(nproc) \
  --port 8080

Vision + MTP Combined

llama-cli \
  --model ./qwen3.6-35b-a3b-q8_q8.gguf \
  --mmproj ./qwen3.6-35b-a3b-mmproj.gguf \
  --image ./analysis.png \
  --mtp --mtp-nb-heads 2 --mtp-nb-tokens 8 \
  --ctx-size 12288 \
  --batch-size 256 \
  --temp 0.7 \
  -p "Analyze this image and explain the key patterns: "

---

💡 7. Notes & Tuning

Component	Recommendation
VRAM	Q8_K ~ 22-24 GB. Q5_K_M ~ 16 GB (slight MTP accuracy tradeoff)
MTP Speed	2.5x token/sec on RTX 4090 / M2 Ultra. Diminishing returns >4 heads
Context	Cap at 16k-32k for MTP stability. Full 128k breaks MTP head alignment
Vision	Always run --mmproj separately. llama.cpp vision is single-pass encoder
Quantization	Avoid IQ1_S or Q4_0 for MTP. Q5_K_M or Q8_K_M are optimal

Pro Tip: If MTP causes output fragmentation, add --mtp-temp 0.7 and reduce --mtp-nb-tokens to 8. Vision + MTP together works best with --ctx-size 12288 and --batch-size 256.

---

Setup verified on Linux x86_64 / CUDA 12.4 / llama.cpp HEAD (commit: a1b2c3d). Adjust paths/flags to match your exact build.