Skip to content

Instantly share code, notes, and snippets.

View timinar's full-sized avatar

Inar TImiryasov timinar

5 followers · 1 following
View GitHub Profile
@timinar
timinar / GLM-5.2-on-8xA100-vLLM.md
Last active August 21, 2026 23:10
Running GLM-5.2 (753B DeepSeek-Sparse-Attention MoE) on 8x A100 80GB with vLLM — TRITON_MLA_SPARSE backend (PR #38476), no-recompile install, benchmarks

Running GLM-5.2 (753B DSA MoE) on 8× A100 80GB with vLLM

TL;DR. GLM-5.2 (glm_moe_dsa — DeepSeek Sparse Attention) does not run on Ampere (A100, sm_80) with stock vLLM: the sparse-MLA attention backend (FLASHMLA_SPARSE) and the lightning-indexer's fp8_mqa_logits (DeepGEMM) are Hopper/Blackwell-only. vLLM PR #38476 (issue #38006) adds a Triton sparse-MLA backend (TRITON_MLA_SPARSE) + a bf16 Triton indexer fallback that run on Ampere. Cherry-picking it onto current main is a Python-only change — no CUDA recompile. Result: GLM-5.2 AWQ-INT4 serves on 8× A100 at ~56 tok/s single-stream and ~625 tok/s aggregate decode (32-way), with coherent output.

This is an independent 8× A100 confirmation of PR #38476 (the author validated on 32× A100), plus a no-recompile install note. Credit to @haosdent for the PR.

Requirements

  • 8× A100 80GB (sm_80). ~410 GiB VRAM used at TP=8, so all 8 GPUs.
@timinar
timinar / gru.py
Created October 10, 2024 12:02
import json
import argparse
from lib.timeseries_dataloader_and_loss_rand import IceCubeDataset, train_validation_loaders
from lib.timeseries_dataloader_and_loss import angular_dist_score_unit_vectors, angles_to_unit_vector
import os
import numpy as np
import torch
import torch.nn as nn