vLLM-Omni AIPCC Standalone Product — Midstream Guidance & Spec

From: Midstream (Doug Smith) For: Srija Ganguly (AIPCC container infra), Andre Lustosa (AIPCC Ecosystems) Date: 2026-07-22 Related tickets: AIPCC-21244, INFERENG-9335, INFERENG-9278, INFERENG-9323

Note: This is our best understanding from the midstream side based on what we currently have. We appreciate Srija's expert input on AIPCC conventions and infrastructure — this is meant to accelerate, not prescribe. If anything conflicts with AIPCC patterns or tooling, AIPCC's conventions win.

Decision: Standalone Product (confirmed July 22)

vllm-omni is a standalone product, following the model-opt precedent — not a variant of RHAIIS.

Rationale:

vllm-omni is a different software stack (vllm + vllm-omni layered), not just vllm on different hardware
It will have its own accelerator variants (CUDA today, ROCm and CPU anticipated)
AIPCC-21244 already scoped standalone infrastructure (own Pyxis, KRD, Tekton)
Confirmed by Andre Lustosa, +1 from Daniele Trifirò

Structural template: collections/model-opt/ — own collection, own wheel package, own build-args directory.

1. Pipeline Collection — `collections/vllm-omni/`

Directory Structure

collections/vllm-omni/cuda13.0-ubi9/
├── requirements.txt
├── constraints.txt
└── constraints-rules.txt

Future variants (not needed for TP): cpu-ubi9/, rocm7.14-ubi9/

`requirements.txt`

Both packages must be pinned together — vllm-omni imports internal vLLM APIs that change between versions (see AIPCC-21515).

# vllm-omni two-wheel strategy: vllm base + vllm-omni on top
# These MUST be version-paired (vllm-omni imports vLLM internals)
vllm[audio,tensorizer,cgraph-cuda12]==<VERSION>+rhaiv.<N>
vllm-omni==<VERSION>+rhaiv.<N>

For reference, our current midstream CUDA build also installs these FlashInfer packages:

# FlashInfer (pre-compiled jit-cache variant for zero cold-start)
# Installed from https://flashinfer.ai/whl/cu130
flashinfer-cubin==0.6.12
flashinfer-jit-cache==0.6.12

`constraints.txt`

Current midstream constraints from neuralmagic/constraints/vllm-omni.txt:

# fastapi 0.137+ changed the routing structure, breaking
# model-hosting-container-standards handler overrides.
fastapi[standard] >= 0.133.0, < 0.137.0

# transformers 5.13.0 changed AutoFeatureExtractor.register() to call
# key.__module__ on the config_class arg, breaking ming.py which passes
# a string. See INFERENG-8926.
transformers >= 5.5.3, < 5.13.0

Anticipated additions (pending v0.25.z rebase, INFERENG-9317):

fa3-fwd removal — this package is being dropped from omni's deps
av version override — specific version pin TBD

We'll update this section once the rebase lands.

`constraints-rules.txt`

torch-2.11.0 *

Same as all other collections — pulls builder-generated constraints from the torch-2.11.0 collection.

2. Containers — `Containerfile.vllm-omni.cuda-ubi9`

Reference: Our Midstream Dockerfile

Our production Dockerfile is in the nm-vllm-omni-ent repo as Dockerfile.ubi. Key characteristics that should carry over:

Two-wheel install order — vLLM base first, then vllm-omni on top:

# The payload/run.sh script handles this:
# 1. Install flashinfer packages (jit-cache)
# 2. Install vLLM wheel with extras: vllm[audio,tensorizer,cgraph-cuda12]
# 3. Install vllm-omni wheel on top (with constraints)
# All installs use --constraints for fastapi/transformers pins

CUDA JIT toolchain — always needed (not just develsdk variant):

# Core CUDA JIT compilation toolchain:
# cuobjdump, nvcc, nvrtc — deepgemm JIT
# cudart-devel — apache-tvm-ffi JIT torch c dlpack extension
RUN microdnf install -y --nodocs \
        cuda-cuobjdump-${CUDA_MAJOR}-${CUDA_MINOR} \
        cuda-nvcc-${CUDA_MAJOR}-${CUDA_MINOR} \
        cuda-nvrtc-${CUDA_MAJOR}-${CUDA_MINOR} \
        cuda-cudart-devel-${CUDA_MAJOR}-${CUDA_MINOR} \
    && microdnf clean all

espeak-ng — built from source (not packaged for UBI 9), required for TTS models (Qwen3-TTS):

FROM base AS espeak-ng-builder
RUN microdnf install -y gcc gcc-c++ make cmake git && \
    git clone --depth 1 --branch 1.52.0 https://github.com/espeak-ng/espeak-ng.git /tmp/espeak-ng && \
    cd /tmp/espeak-ng && \
    cmake -B build -DCMAKE_INSTALL_PREFIX=/usr/local -DUSE_AUDIO=OFF -DBUILD_SHARED_LIBS=OFF && \
    cmake --build build -j$(nproc) && \
    cmake --install build

# In the release stage:
COPY --from=espeak-ng-builder /usr/local/bin/espeak-ng /usr/local/bin/
COPY --from=espeak-ng-builder /usr/local/share/espeak-ng-data /usr/local/share/espeak-ng-data

ffmpeg — we use imageio-ffmpeg's bundled static binary (pip-installed, no RPM):

# ffmpeg shim: symlink imageio-ffmpeg's bundled static binary into PATH
# so model pipelines that shell out to `ffmpeg` can find it without
# pulling ffmpeg from an external RPM repo.
COPY midstream/install-ffmpeg-shim.py /tmp/
RUN python /tmp/install-ffmpeg-shim.py && rm /tmp/install-ffmpeg-shim.py

Triton — needs gcc and git at runtime:

RUN microdnf install -y gcc git && microdnf clean all

Entrypoint:

# vllm-omni uses vllm entrypoint with --omni flag (matches upstream pattern)
CMD ["vllm", "serve", "--omni"]

Environment variables:

ENV HF_HUB_OFFLINE=1 \
    HOME=/home/vllm \
    VLLM_USAGE_STATS_SERVER=https://console.redhat.com/api/rhaiis-stats \
    VLLM_USAGE_SOURCE=quay-docker-image \
    OUTLINES_CACHE_DIR=/tmp/outlines \
    NUMBA_CACHE_DIR=/tmp/numba \
    TRITON_CACHE_DIR=/tmp/triton \
    VLLM_CACHE_ROOT=/tmp/vllm \
    FLASHINFER_WORKSPACE_BASE=/tmp

OpenShift compatibility:

RUN umask 0002 \
    && mkdir -p /licenses /home/vllm/.cache \
    && chmod g+rwx /home/vllm /home/vllm/.cache

Differences from Standard RHAIIS Containerfile

Feature	Standard RHAIIS	vllm-omni
Entrypoint	`python3 -m vllm.entrypoints.openai.api_server`	`vllm serve --omni`
Wheel install	Single vllm wheel	Two wheels: vllm + vllm-omni (order matters)
espeak-ng	Not needed	Required (multi-stage build from source)
ffmpeg	Not needed	Required (imageio-ffmpeg shim)
Tiktoken pre-download	Yes (chat templates)	TBD — may not be needed for omni models
NCCL config	Yes (`COPY nccl_configfile`)	Yes (same multi-GPU needs)
Chat templates	`COPY template/*.jinja`	TBD — omni models may use different templates
Constraints	None applied during install	`--constraints` flag on all `uv pip install` calls
scanlibs ignores	flashinfer, cupy, torio	flashinfer, cupy, torio (same set, possibly more for omni deps)

3. Build-Args — `build-args.vllm-omni/`

Following model-opt's pattern of a separate build-args directory:

`build-args.vllm-omni/cuda-ubi9.conf`

WHEEL_RELEASE_PACKAGE=vllm-omni-wheels
WHEEL_RELEASE_<version fields TBD by AIPCC convention>
BASE_IMAGE=<same CUDA base image as standard RHAIIS>

4. Labels & Product Identity

Following standalone pattern:

Field	Value
Container label	`rhaiis-vllm-omni-cuda-rhel9-container` (suggested)
Product name	"Red Hat AI Inference Server for Multimodal" or similar

Key Technical Details

Two-Wheel Version Pairing (Critical)

vllm-omni imports vLLM internals (from vllm.utils.torch_utils import supports_xccl, etc.). If the versions aren't paired, you get import crashes. This is the root cause of AIPCC-21515.

In the pipeline requirements.txt, both packages must pin to matching versions. Our midstream CI enforces this by building both wheels from coordinated branches and installing them in sequence.

Install Order

Must be: flashinfer → vLLM → vllm-omni. The vllm-omni wheel depends on vllm being installed first. Our prepare.sh generates a payload/run.sh that handles this ordering — AIPCC's install-wheel-release.sh will need equivalent logic.

System Dependencies Beyond Python

Dependency	Why	How We Install It
espeak-ng 1.52.0	TTS phonemizer backend (Qwen3-TTS)	Built from source in multi-stage Dockerfile
ffmpeg (static binary)	Audio/video codec for multimodal pipelines	`imageio-ffmpeg` pip package + PATH shim script
gcc, git	Triton JIT compilation, triton-kernels git install	`microdnf install`
CUDA cuobjdump, nvcc, nvrtc, cudart-devel	deepgemm JIT, tvm-ffi JIT	`microdnf install` from CUDA repo

Constraints Applied During Install

All uv pip install calls in our build use --constraints pointing at the constraints file. This prevents pip from pulling a too-new fastapi or transformers that would break at runtime. The AIPCC equivalent should apply constraints.txt during the wheel collection install step.

What's Coming (Not Blocking TP)

fa3-fwd removal — being dropped from omni deps in v0.25.z rebase (INFERENG-9317)
av version override — specific pin TBD from same rebase
Additional accelerator variants — ROCm, CPU (post-TP, we have midstream Dockerfiles for both)
Model template refinement — chat templates and tiktoken may differ from standard RHAIIS

Contacts

Person	Role
Doug Smith (midstream)	Constraints, build pipeline, Dockerfiles
Daniele Trifirò (midstream)	AIPCC architecture SME
Tarun Kumar (midstream)	AIPCC integration, model validation
Ricardo Noriega (OCTO)	Coordination lead

dougbtv/vllm-omni-aipcc-spec.md

Select an option

No results found

Select an option

No results found

vLLM-Omni AIPCC Standalone Product — Midstream Guidance & Spec

Decision: Standalone Product (confirmed July 22)

1. Pipeline Collection — `collections/vllm-omni/`

Directory Structure

`requirements.txt`

`constraints.txt`

`constraints-rules.txt`

2. Containers — `Containerfile.vllm-omni.cuda-ubi9`

Reference: Our Midstream Dockerfile

Differences from Standard RHAIIS Containerfile

3. Build-Args — `build-args.vllm-omni/`

`build-args.vllm-omni/cuda-ubi9.conf`

4. Labels & Product Identity

Key Technical Details

Two-Wheel Version Pairing (Critical)

Install Order

System Dependencies Beyond Python

Constraints Applied During Install

What's Coming (Not Blocking TP)

Contacts

dougbtv/vllm-omni-aipcc-spec.md

vLLM-Omni AIPCC Standalone Product — Midstream Guidance & Spec

Decision: Standalone Product (confirmed July 22)

1. Pipeline Collection — collections/vllm-omni/

Directory Structure

requirements.txt

constraints.txt

constraints-rules.txt

2. Containers — Containerfile.vllm-omni.cuda-ubi9

Reference: Our Midstream Dockerfile

Differences from Standard RHAIIS Containerfile

3. Build-Args — build-args.vllm-omni/

build-args.vllm-omni/cuda-ubi9.conf

4. Labels & Product Identity

Key Technical Details

Two-Wheel Version Pairing (Critical)

Install Order

System Dependencies Beyond Python

Constraints Applied During Install

What's Coming (Not Blocking TP)

Contacts

1. Pipeline Collection — `collections/vllm-omni/`

`requirements.txt`

`constraints.txt`

`constraints-rules.txt`

2. Containers — `Containerfile.vllm-omni.cuda-ubi9`

3. Build-Args — `build-args.vllm-omni/`

`build-args.vllm-omni/cuda-ubi9.conf`