Blaizzy

Qwen3.8 fine-grained FP8 → native MLX MXFP8 support

Complete proposed changes for loading, converting, and testing Qwen/Qwen3.8-27B-FP8 in mlx-vlm, based on Blaizzy/mlx-vlm commit 74b740b9bb1a1248e4b7b12b3b1ba7cae9c21e9f.

Why this is needed

The Qwen checkpoint stores projection weights as E4M3 bytes with arbitrary BF16 inverse scales on a 128×128 block grid. Native MLX MXFP8 instead uses E8M0 scales per output row and per 32-value group.

DeepSeek V4 can losslessly repack its FP8 tensors because its source scales are already UE8M0-compatible. Qwen cannot use that byte reinterpretation. Its source blocks must be reconstructed and immediately requantized into native MLX MXFP8.

Blaizzy

diff --git a/mlx_vlm/models/rope_utils.py b/mlx_vlm/models/rope_utils.py
index 0d2f5343..6fb292ae 100644
--- a/mlx_vlm/models/rope_utils.py
+++ b/mlx_vlm/models/rope_utils.py
@@ -556,6 +556,7 @@ class MRoPERotaryEmbedding(nn.Module):
         self.pairing = _pairing_for_style(style)
         self.fused_apply = self.position_selector is not None and _HAS_METAL
         self._compiled_apply = {} if self.fused_apply else None
+        self.eval_cached_arrays()
 

Blaizzy

"""
Benchmark TriAttention on MATH 500 — matching the paper's evaluation protocol.

Paper settings: max_tokens=32768, temp=0.6, top_p=0.95, budget=512/1024/2048
We use max_tokens=4096 for practical runtime on Apple Silicon.

USAGE
    python bench_triattention_math.py \
        --model /tmp/gemma-4-26b-a4b-it-5bit \
        --calib /tmp/gemma4_26b_5bit_calib.safetensors \

Blaizzy

"""
Benchmark TurboQuant (TBQ) vs baseline on MM-NIAH (Multimodal Needle-in-a-Haystack).

INSTALL
    pip install -U mlx-vlm
    # or
    uv pip install -U mlx-vlm

SETUP — Extract images (one-time)
    huggingface-cli download OpenGVLab/MM-NIAH mm_niah_val/images.tar.gz --repo-type dataset

Blaizzy

"""Benchmark TurboQuant vs baseline on LongBench-v2.

Usage:
    python scripts/bench_longbench_v2.py --model google/gemma-4-e4b-it --num-samples 10 --max-tokens-ctx 260000
    python scripts/bench_longbench_v2.py --model google/gemma-4-26b-a4b-it --num-samples 5 --max-tokens-ctx 128000 --kv-bits 4
"""

import argparse
import importlib
import time

Blaizzy

#!/usr/bin/env python3
"""
Benchmark for Qwen3-TTS: measures TTFB, inter-chunk latency, and throughput.

Usage:                                                                                                                                                                                
  # Sequential only (short/medium/long)                                                                                                                                                                 
  python qwen3_tts_benchmark.py --model mlx-community/Qwen3-TTS-12Hz-0.6B-Base-bf16

  # Sequential + batched (1,2,3,4,8)
  python qwen3_tts_benchmark.py --batch-size 1 2 3 4 8

Blaizzy

#!/usr/bin/env python3
"""
Benchmark for Qwen3-TTS: measures TTFB, inter-chunk latency, and throughput.

Usage:
    python benchmarks/qwen3_tts_benchmark.py
    python benchmarks/qwen3_tts_benchmark.py --model mlx-community/Qwen3-TTS-0.6B-bf16
    python benchmarks/qwen3_tts_benchmark.py --num-trials 3 --streaming-interval 1.0
    python benchmarks/qwen3_tts_benchmark.py --prompts short medium long
"""

Blaizzy

Qwen3-TTS Batch Generation Benchmark Results

Voice: serena Device: Apple Silicon (MLX) Date: 2025-02-28

---

Cross-Model Comparison (batch=4, short prompt)

Blaizzy

//
//  BaseConfiguration.swift
//  mlx-test
//
//  Created by Prince Canuma on 29/12/25.
//

import Foundation
import MLX

Blaizzy

mlx_audio.tts.generate \
  --model mlx-community/chatterbox-turbo-fp16 \
  --text 'Abstract
The dominant sequence transduction models are based on complex recurrent or
convolutional neural networks that include an encoder and a decoder. The best
performing models also connect the encoder and decoder through an attention
mechanism. We propose a new simple network architecture, the Transformer,
based solely on attention mechanisms, dispensing with recurrence and convolutions
entirely. Experiments on two machine translation tasks show these models to
be superior in quality while being more parallelizable and requiring significantly