Smoke test of the Qwen 3.8 Max model (2.4T total parameters, 95B active) using community NVFP4 weights on 8x NVIDIA B300 SXM6 275GB GPUs, served via the midstream v0.27.1 container image.
| Field | Value |
|---|---|
| Model | Inferact/Qwen3.8-2.4T-A95B-NVFP4 (community NVFP4 quant of Qwen/Qwen3.8-2.4T-A95B) |
| Container Image | quay.io/vllm/automation-vllm:cuda-31490750632 |
A guide for serving NVIDIA's Nemotron 3.5 Lightning model using the Red Hat AI Inference Server with vLLM.
Nemotron 3.5 Lightning is a 30B parameter Mixture-of-Experts model with 3B parameters active per forward pass (A3B). The NVFP4 variant uses NVIDIA's ModelOpt mixed quantization (NVFP4 weights + MXFP8 + FP8 E4M3 KV cache), bringing the on-disk footprint down to ~20 GiB.
The model includes reasoning capabilities — chat responses include a thinking/reasoning trace before the final answer.
Epic: INFERENG-9476 From: Doug Smith Date: 2026-07-29
From: Midstream (Doug Smith) For: Srija Ganguly (AIPCC container infra), Andre Lustosa (AIPCC Ecosystems) Date: 2026-07-22 Related tickets: AIPCC-21244, INFERENG-9335, INFERENG-9278, INFERENG-9323
Note: This is our best understanding from the midstream side based on what we currently have. We appreciate Srija's expert input on AIPCC conventions and infrastructure — this is meant to accelerate, not prescribe. If anything conflicts with AIPCC patterns or tooling, AIPCC's
A guide for serving Poolside's Laguna S 2.1 model using the Red Hat AI Inference Server with vLLM.
Laguna S 2.1 is a 118B parameter Mixture-of-Experts agentic coding model with 8B parameters active per forward pass. It uses 256 experts with top-10 routing across 48 layers (36 sliding-window + 12 global attention in a 3:1 ratio) and supports up to 256K context length.
The FP8 quantized version weighs ~121 GB on disk and fits comfortably on 4x H100 80GB GPUs.
Codename Arcee is a day-0 preview release. Open weights go live Sunday, July 27th. Doug is on PTO Friday July 25 through Monday July 28, so this needs to be picked up by someone else on the team.
This doc covers what's already done, what needs to happen, and where to find the detailed playbooks.
The gist is...
Inkling (975B param, 41B active MoE) serves on 8x H200 using the thinkingmachines/Inkling-NVFP4 checkpoint. The upstream PRs target Blackwell (SM100), so three patches are needed for Hopper (SM90).
All patches live on doug/inkling-0day (based on the upstream Add Inkling multi-depth MTP support [5/N] PR merged onto our v0.25.0 base).
The Inkling NVFP4 model is running on 8x NVIDIA B200 GPUs, served via the Red Hat AI Inference Server (vLLM). The server exposes an OpenAI-compatible API on port 8000.
SSH into the box and run queries from inside the container (podman pasta networking means host curl won't work — use podman exec instead):
ssh dougbtv@10.14.216.231
This guide covers running the Red Hat AI Inference Server to serve Thinking Machines' Inkling model, powered by vLLM.
Inkling is a 975B parameter Mixture-of-Experts model from Thinking Machines — 41B parameters active per forward pass, which keeps it fast despite the massive total size. It's multimodal (text + vision), natively supports tool use, and ships with strong multilingual and reasoning capabilities out of the box.
The NVFP4 quantized variant fits on 8 GPUs and weighs in at ~552GB on disk, which is surprisingly manageable for a model this big. If you've served DeepSeek-V3 or similar large MoE models before, this is the same general ballpark.
Serving Thinking Machines Inkling (975B MoE, 41B active) with vLLM on 8x H200 GPUs.