Inkling NVFP4 on B200 — demo queries for Sawyer

inkling-demo-for-sawyer.md

Inkling NVFP4 — Live on B200 (dgx-b200-02)

The Inkling NVFP4 model is running on 8x NVIDIA B200 GPUs, served via the Red Hat AI Inference Server (vLLM). The server exposes an OpenAI-compatible API on port 8000.

Access

SSH into the box and run queries from inside the container (podman pasta networking means host curl won't work — use podman exec instead):

ssh dougbtv@10.14.216.231

Or, set up a local tunnel so you can hit localhost:8000 from your own machine:

ssh -L 8000:localhost:8000 dougbtv@10.14.216.231

With the tunnel active, all the curl examples below work directly from your laptop (just drop the podman exec vllm-inkling prefix).

Health Check

podman exec vllm-inkling curl -s http://localhost:8000/health

Text Chat

podman exec vllm-inkling curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "thinkingmachines/Inkling-NVFP4",
    "messages": [
      {"role": "user", "content": "Write a haiku about distributed GPU inference."}
    ],
    "max_tokens": 128
  }' | python3 -m json.tool

Multimodal (Vision)

Inkling supports image inputs — pass an image URL alongside your text prompt:

podman exec vllm-inkling curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "thinkingmachines/Inkling-NVFP4",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/a/a7/Camponotus_flavomarginatus_ant.jpg/640px-Camponotus_flavomarginatus_ant.jpg"}},
          {"type": "text", "text": "What is in this image? Describe it in detail."}
        ]
      }
    ],
    "max_tokens": 256
  }' | python3 -m json.tool

Longer Generation

podman exec vllm-inkling curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "thinkingmachines/Inkling-NVFP4",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Explain what a Mixture-of-Experts model is and why it matters for large-scale AI inference. Keep it approachable for a non-technical audience."}
    ],
    "max_tokens": 512
  }' | python3 -m json.tool

Model Info

• Model: thinkingmachines/Inkling-NVFP4
• Parameters: 975B total, 41B active (Mixture-of-Experts)
• Quantization: NVFP4 (~552 GB on disk)
• GPUs: 8x NVIDIA B200 (SM100, Blackwell)
• Serving: Red Hat AI Inference Server (vLLM) — quay.io/vllm/automation-vllm:cuda-29455473553

Tips

• Pretty-print responses: pipe output through python3 -m json.tool or jq .
• Extract just the text: ... | jq -r '.choices[0].message.content'
• Streaming: add "stream": true to the request body for token-by-token output
• Context length: server is configured with --max-model-len 8192