RHAII Early Access: Inkling (Thinking Machines 975B MoE)

inkling-early-access.md

RHAII Early Access: Inkling

This guide covers running the Red Hat AI Inference Server to serve Thinking Machines' Inkling model, powered by vLLM.

About Inkling

Inkling is a 975B parameter Mixture-of-Experts model from Thinking Machines — 41B parameters active per forward pass, which keeps it fast despite the massive total size. It's multimodal (text + vision), natively supports tool use, and ships with strong multilingual and reasoning capabilities out of the box.

The NVFP4 quantized variant fits on 8 GPUs and weighs in at ~552GB on disk, which is surprisingly manageable for a model this big. If you've served DeepSeek-V3 or similar large MoE models before, this is the same general ballpark.

• Model card: thinkingmachines/Inkling-NVFP4
• Red Hat container catalog: rhaii-preview/vllm-cuda-rhel9

Prerequisites

• 8x NVIDIA H200 or B200 GPUs
• ~552 GB disk for model weights
• Podman with NVIDIA CDI support
• A Hugging Face account (the model is gated — you'll need to accept the license on the model card first)

Download the Model

HF_HUB_CACHE=/path/to/hub_cache \
  huggingface-cli download thinkingmachines/Inkling-NVFP4

This takes a while (~552 GB). Grab a coffee, catch up on your favorite subreddit, whatever you have to do.

Pull the Image

podman pull registry.redhat.io/rhaii-preview/vllm-cuda-rhel9:inkling

Start the Server

podman run -d \
  --name vllm-inkling \
  --device nvidia.com/gpu=all \
  --ipc=host \
  --security-opt=label=disable \
  -p 8000:8000 \
  -v /path/to/hub_cache:/hf:Z \
  -e HF_HUB_CACHE=/hf \
  -e HF_HUB_OFFLINE=1 \
  registry.redhat.io/rhaii-preview/vllm-cuda-rhel9:inkling \
    --model thinkingmachines/Inkling-NVFP4 \
    --tensor-parallel-size 8 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.90 \
    --trust-remote-code \
    --host 0.0.0.0 --port 8000

Startup takes a few minutes — the model needs to load weights across all 8 GPUs and warm up CUDA graphs. On B200 (SM100), there's also a flashinfer autotuning pass and CuTeDSL kernel compilation step that adds a couple minutes.

Watch the logs with:

podman logs -f vllm-inkling

You're good to go when you see Application startup complete.

Try It Out

Health check

curl http://localhost:8000/health

Chat completion

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "thinkingmachines/Inkling-NVFP4",
    "messages": [
      {"role": "user", "content": "Write a perl script that generates poems about CI failures with metaphors about east coast sandwiches such as the chopped cheese and the philly cheese steak"}
    ],
    "max_tokens": 512
  }'

Something a little more practical

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "thinkingmachines/Inkling-NVFP4",
    "messages": [
      {"role": "user", "content": "Explain the tradeoffs between tensor parallelism and pipeline parallelism for serving large MoE models. Keep it under 200 words."}
    ],
    "max_tokens": 256
  }'

Completions endpoint

curl -s http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "thinkingmachines/Inkling-NVFP4",
    "prompt": "Three reasons to ski the east coast instead of out west:\n1.",
    "max_tokens": 256
  }'

Cleanup

podman stop vllm-inkling
podman rm vllm-inkling

Notes

• HF_HUB_CACHE not HF_HOME — the model resolves from the cache directory directly. Using HF_HOME will cause a "model not found" error since it expects a hub/ subdirectory.
• --ipc=host — required for shared memory across TP workers. Don't combine this with --shm-size (they conflict in podman).
• Startup time — expect 5-10 minutes for full warmup depending on GPU type. The model loads fast but CUDA graph capture takes a bit with 8-way TP.
• Context length — 8192 is a safe starting point. The model supports longer contexts but you'll need to tune --gpu-memory-utilization and batch size accordingly.