From: Midstream (Doug Smith) For: Srija Ganguly (AIPCC container infra), Andre Lustosa (AIPCC Ecosystems) Date: 2026-07-22 Related tickets: AIPCC-21244, INFERENG-9335, INFERENG-9278, INFERENG-9323
Note: This is our best understanding from the midstream side based on what we currently have. We appreciate Srija's expert input on AIPCC conventions and infrastructure — this is meant to accelerate, not prescribe. If anything conflicts with AIPCC patterns or tooling, AIPCC's
A guide for serving Poolside's Laguna S 2.1 model using the Red Hat AI Inference Server with vLLM.
Laguna S 2.1 is a 118B parameter Mixture-of-Experts agentic coding model with 8B parameters active per forward pass. It uses 256 experts with top-10 routing across 48 layers (36 sliding-window + 12 global attention in a 3:1 ratio) and supports up to 256K context length.
The FP8 quantized version weighs ~121 GB on disk and fits comfortably on 4x H100 80GB GPUs.
Codename Arcee is a day-0 preview release. Open weights go live Sunday, July 27th. Doug is on PTO Friday July 25 through Monday July 28, so this needs to be picked up by someone else on the team.
This doc covers what's already done, what needs to happen, and where to find the detailed playbooks.
| When | What |
|---|
Inkling (975B param, 41B active MoE) serves on 8x H200 using the thinkingmachines/Inkling-NVFP4 checkpoint. The upstream PRs target Blackwell (SM100), so three patches are needed for Hopper (SM90).
All patches live on doug/inkling-0day (based on the upstream Add Inkling multi-depth MTP support [5/N] PR merged onto our v0.25.0 base).
The Inkling NVFP4 model is running on 8x NVIDIA B200 GPUs, served via the Red Hat AI Inference Server (vLLM). The server exposes an OpenAI-compatible API on port 8000.
SSH into the box and run queries from inside the container (podman pasta networking means host curl won't work — use podman exec instead):
ssh dougbtv@10.14.216.231
This guide covers running the Red Hat AI Inference Server to serve Thinking Machines' Inkling model, powered by vLLM.
Inkling is a 975B parameter Mixture-of-Experts model from Thinking Machines — 41B parameters active per forward pass, which keeps it fast despite the massive total size. It's multimodal (text + vision), natively supports tool use, and ships with strong multilingual and reasoning capabilities out of the box.
The NVFP4 quantized variant fits on 8 GPUs and weighs in at ~552GB on disk, which is surprisingly manageable for a model this big. If you've served DeepSeek-V3 or similar large MoE models before, this is the same general ballpark.
Serving Thinking Machines Inkling (975B MoE, 41B active) with vLLM on 8x H200 GPUs.
| Field | Value |
|---|---|
| Model | google/gemma-4-12B-it — encoder-free multimodal dense 12B (text, image, audio) |
| Early Access Image | quay.io/vllm/rhaiis-early-access:gemma4-unified-qat |
| Build Run | 27027630533 |
| nm-vllm-ent branch | doug/kup-0day-nightly |
Deploy and test nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4 — a 550B MoE frontier model (55B active parameters) in NVFP4 quantization — using nm-vllm-ent based on upstream vLLM v0.22.1 on NVIDIA H200 GPUs.
| Field | Value |
|---|
| Field | Value |
|---|---|
| Models | gemma-4-E4B-it-qat-mobile-ct, gemma-4-E2B-it-qat-mobile-ct (QAT), google/gemma-4-12B-it (Unified) |
| Early Access Image | quay.io/vllm/rhaiis-early-access:gemma4-unified-qat |
| Build Run (Nightly V2) | 27027630533 |
| nm-vllm-ent branch | doug/kup-0day-nightly |