Gemma 4 Unified (12B): Build, Run, and Smoke Test Guide (nm-vllm-ent)

gemma4-unified-howto.md

Gemma 4 Unified (12B): Build, Run, and Smoke Test Guide (nm-vllm-ent)

Build Info

Field	Value
Model	google/gemma-4-12B-it — encoder-free multimodal dense 12B (text, image, audio)
Early Access Image	quay.io/vllm/rhaiis-early-access:gemma4-unified-qat
Build Run	27027630533
nm-vllm-ent branch	doug/kup-0day-nightly
nm-cicd branch	doug/kup-0day-nightly
Base	doug/v0.22.0 + upstream/main + PR #44340 + PR #44429 + #44571
Target device	cuda
CUDA	13.0
Python	3.12

What's in the branch

• upstream/main merge — includes Gemma 4 Unified support (#44429) and vision embedder quantization fix (#44571)
• PR #44340 — compressed-tensors WNA8O8Int + WNInt embeddings (also supports GemmaQAT models on the same image)
• ParallelLMHead fix — getattr chain fallback for embedding-based layers in humming_utils.py

Upstream resources

• Deployment recipe: recipes.vllm.ai/Google/gemma-4-12B-it
• Upstream PR: #44429 (merged)
• Architecture: Gemma4UnifiedForConditionalGeneration — encoder-free multimodal (no separate vision/audio towers)

How to Run

Prerequisites

• 1x A100 80GB or H200 GPU (~24 GiB bf16 model, fits on a single GPU)
• Model weights from HuggingFace: google/gemma-4-12B-it
• podman with NVIDIA CDI support

Download the model

HF_HUB_CACHE=/path/to/hub_cache \
  uv tool run --from huggingface_hub hf download google/gemma-4-12B-it

Pull the image

podman pull quay.io/vllm/rhaiis-early-access:gemma4-unified-qat

Start the server

podman run -d \
  --name vllm-gemma4-unified \
  --user root \
  --device nvidia.com/gpu=0 \
  --security-opt=label=disable \
  --shm-size=10g \
  -p 8000:8000 \
  -v /path/to/hub_cache:/hf/hub:Z \
  -e FLASHINFER_DISABLE_VERSION_CHECK=1 \
  -e CUDA_VISIBLE_DEVICES=0 \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e HF_HOME=/hf \
  --entrypoint bash \
  quay.io/vllm/rhaiis-early-access:gemma4-unified-qat \
  -c "ln -sf /opt/vllm/lib/python3.12/site-packages/nvidia/cu13/lib/libnvrtc.so.13 \
      /opt/vllm/lib/python3.12/site-packages/nvidia/cu13/lib/libnvrtc.so ; \
  python3 -m vllm.entrypoints.openai.api_server \
    --model google/gemma-4-12B-it \
    --gpu-memory-utilization 0.9 \
    --trust-remote-code \
    --enforce-eager \
    --limit-mm-per-prompt '{\"image\":1,\"audio\":1}' \
    --host 0.0.0.0 --port 8000"

Notes:

• Mount your HF cache as /hf/hub (not /hf) — the image sets HF_HUB_OFFLINE=1 so the model must be pre-downloaded.
• --user root + the ln -sf workaround is needed for the libnvrtc symlink.
• --limit-mm-per-prompt '{"image":1,"audio":1}' enables multimodal input.
• --enforce-eager is recommended for reliable startup.
• Total boot ~2.5 min on H200.

Watch startup logs

podman logs -f vllm-gemma4-unified

Look for Application startup complete.

Smoke Test

Health check

curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8000/health
# Expected: 200

Text (chat completions)

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-12B-it",
    "messages": [{"role": "user", "content": "What is the US state capital with the smallest population? One word."}],
    "max_tokens": 10,
    "temperature": 0
  }'

Expected: "Montpelier"

Audio (transcription)

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-12B-it",
    "messages": [{"role": "user", "content": [
      {"type": "audio_url", "audio_url": {"url": "https://github.com/track-forge/snippets/raw/main/tt-weasels.ogg"}},
      {"type": "text", "text": "Transcribe the speech word for word."}
    ]}],
    "max_tokens": 60,
    "temperature": 0
  }'

Expected: Something about weasels eating the phone system.

Vision (image)

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-12B-it",
    "messages": [{"role": "user", "content": [
      {"type": "image_url", "image_url": {"url": "http://images.cocodataset.org/val2017/000000039769.jpg"}},
      {"type": "text", "text": "What animals are in this image?"}
    ]}],
    "max_tokens": 40,
    "temperature": 0
  }'

Cleanup

podman stop vllm-gemma4-unified
podman rm vllm-gemma4-unified

Related

• GemmaQAT Howto Gist — same image also supports GemmaQAT E4B/E2B pre-release models

JIRA

• INFERENG-7479 — Build Wheel & Container Image
• INFERENG-7574 — Gemma 4 Unified experimental build