Skip to content

Instantly share code, notes, and snippets.

@dougbtv

dougbtv/arcee-handoff-draft.md

Last active July 23, 2026 14:45
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save dougbtv/705a72c68da8266c3f4072618f6d4cdd to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save dougbtv/705a72c68da8266c3f4072618f6d4cdd to your computer and use it in GitHub Desktop.
Download ZIP
Arcee 0-Day Handoff Guide — midstream preview release for July 27th
Raw
arcee-handoff-draft.md

Arcee 0-Day Handoff Guide

Codename Arcee is a day-0 preview release. Open weights go live Sunday, July 27th. Doug is on PTO Friday July 25 through Monday July 28, so this needs to be picked up by someone else on the team.

This doc covers what's already done, what needs to happen, and where to find the detailed playbooks.

Human tl;dr

The gist is...

  • Cherry pick / merge upstream commits into a new branch in midstream (like a targeted sync)
  • Make a CUDA build of the branch using nm-cicd (and --ref of a nm-cicd branch), e.g. using build-whl-image
  • Validate that the build works, deploy it on a box with the new model weights
  • When it's validated, tag it as quay.io/vllm/rhaiis-early-access:kimi-k3
  • Notify AIPCC People (@klara @jrusz) in #day-0-squad in internal RH slack.

Timeline

When What
Thursday July 24 Doug pre-builds a midstream baseline from upstream main
Monday July 27 Open weights go live, upstream vLLM support PR drops
Monday on drop Whoever picks this up merges the model support, builds, tests, and hands off to AIPCC

JIRA Stories

All stories are under epic INFERENG-9289.

Pick up and work this one — it's the core build work:

  • INFERENG-9291 — Arcee: Build Wheel & Container Image — Merge, build, deploy, smoke test. Post build tracking comments here as you iterate (see the build table format in the Inkling example above). Assign yourself and move to In Progress when you start.

Pick up if/when the image is validated and ready for AIPCC:

  • INFERENG-9293 — Arcee: Image Publishing — Tag the image, push to early-access, notify jrusz and Klara.

Pick up if marketing requests comms material:

  • INFERENG-9294 — Arcee: Documentation & Comms — Our usual methodology is to publish a GitHub gist with the build info, deploy steps, smoke test commands, and gotchas (see the skill files for the template and this Gemma 4 example for the expected format). The gist serves as both a technical reference and source material for marketing.

Using the Skill Docs

The full 0-day playbook — merge strategies, build dispatch, runner selection, dev box deployment, smoke testing, AIPCC handoff, image labeling, gotchas — is documented in two skill files:

  • midstream-0-day-release/SKILL.md — Full 0-day lifecycle from notification through AIPCC handoff and comms
  • midstream-build-from-upstream/SKILL.md — Merge strategies, GH Actions build dispatch, dev box deployment, smoke testing

These are available in the rhaiis-midstream-snippets repo under plugins/midstream/skills/.

Recommended approach: Use Claude Code (or your favorite agent harness) and point it at these skill files. They contain the complete build/deploy/test cycle with all the edge cases and lessons learned from previous releases. The agent can drive the build monitoring, smoke testing, and JIRA updates autonomously — you provide the merge inputs and hardware access.

Previous 0-day builds for reference

These JIRA stories have the full comment trail from recent 0-day builds. Read through them to see how the build iterations, troubleshooting, and handoff played out in practice:

  • INFERENG-9108 — Inkling: Build Wheel & Container Image (14 comments, detailed build iteration trail)
  • INFERENG-8139 — Blurr: Build Wheel & Container Image

What's Already Done

  • JIRA epic: INFERENG-9289 — stories created (see JIRA section below)
  • Baseline branch: Doug will push a fresh doug/arcee branch to neuralmagic/nm-vllm-ent on Thursday, built from upstream main. A matching doug/arcee branch will exist in neuralmagic/nm-cicd.
  • Upstream research: KDA, Kimi Linear, and FusedMoE foundations are already on vLLM main. The likely new pieces are AttnRes (Attention Residuals) and Stable LatentMoE support. See the comment on INFERENG-9291 for the full analysis.

What You Need To Do

1. Get the cherry-pick list from Michael Goin

mgoin confirmed that Inferact is building the vLLM integration and it will be dropped publicly on day 0. When the code appears:

  • mgoin will provide instructions on what to cherry-pick (PRs, commits, or a branch)
  • The support code may land as a single PR to vllm-project/vllm, or as commits in a private fork that get pushed publicly on release day

2. Merge model support into the midstream branch

Starting from the doug/arcee baseline branch:

If mgoin provides specific PRs/commits to cherry-pick:

cd /path/to/nm-vllm-ent
git checkout doug/arcee
git fetch https://github.com/<author>/<repo>.git <branch>
PR_BASE=$(git merge-base FETCH_HEAD upstream/main)
git checkout -b pr-k3-clean $PR_BASE
git merge FETCH_HEAD --no-edit
git checkout doug/arcee
git merge -X theirs pr-k3-clean --no-edit
git branch -D pr-k3-clean
git rm -rf .github/ 2>/dev/null && git commit -m "Remove upstream .github directory"
git push origin doug/arcee

If a full sync from main is needed (likely, if the PR targets latest main):

cd /path/to/nm-vllm-ent
git checkout doug/arcee
git fetch upstream main
git merge -X theirs upstream/main --no-edit
git rm -rf .github/ 2>/dev/null && git commit -m "Remove upstream .github directory"
git push origin doug/arcee

Then cherry-pick the model-specific commits on top.

3. Build the image

Dispatch a full wheel + image build:

gh workflow run build-whl-image.yml \
  --repo neuralmagic/nm-cicd \
  --ref doug/arcee \
  -f repo=neuralmagic/nm-vllm-ent \
  -f branch=doug/arcee \
  -f build_label=k8s-a100-build-12-9 \
  -f build_timeout=120 \
  -f image_label=k8s-a100-util \
  -f python=3.12 \
  -f release_image=false \
  -f target_device=cuda

Note: Check the upstream Dockerfile's default CUDA version before dispatching. If upstream has moved to CUDA 13.0, use k8s-a100-build-13-0 for the build label instead.

Monitor the build:

# Get run ID (wait a few seconds after dispatch)
gh run list --repo neuralmagic/nm-cicd --workflow build-whl-image.yml --limit 3 \
  --json databaseId,headBranch,status,conclusion,displayTitle

# Watch progress
gh run view <RUN_ID> --repo neuralmagic/nm-cicd --json status,conclusion,jobs \
  --jq '{status: .status, conclusion: .conclusion, jobs: [.jobs[] | {name: .name, status: .status, conclusion: .conclusion}]}'

On success, extract the image tags:

# Get image build job ID
gh api repos/neuralmagic/nm-cicd/actions/runs/<RUN_ID>/jobs \
  --jq '.jobs[] | select(.name | contains("image")) | .id'

# Extract image tags
gh api repos/neuralmagic/nm-cicd/actions/jobs/<JOB_ID>/logs 2>&1 \
  | sed 's/\x1b\[[0-9;]*m//g' | grep "naming to quay"

4. Smoke test on a dev box

This model is massive (needs a full B300 node or multi-node B200). Confirm hardware availability with mgoin before attempting deployment. Standard smoke test flow:

# Reserve GPUs, pull image, start container
# Health check
curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:8000/health
# Chat completion
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "<MODEL_ID>", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 128}'

Start with --enforce-eager and --max-model-len 4096 for initial testing.

5. Tag and hand off to AIPCC

Once you have a validated image:

# Tag for early access
podman tag quay.io/vllm/automation-vllm:cuda-<run-id> \
  quay.io/vllm/rhaiis-early-access:kimi-k3
podman push quay.io/vllm/rhaiis-early-access:kimi-k3

Notify AIPCC with:

  1. Image: quay.io/vllm/rhaiis-early-access:kimi-k3
  2. LABEL text: rhaiis.preview=Kimi K3
  3. Branch: doug/arcee on nm-vllm-ent
  4. Constraints: hardware limitations, tested configurations

AIPCC contacts: Jakub Rusz (jrusz) and Klara

Key Contacts

Who Role
Michael Goin (mgoin) Hardware sizing, upstream integration status, cherry-pick guidance
Rob Greenberg Initial heads-up, external comms
Jakub Rusz (jrusz) AIPCC handoff, Konflux build
Klara AIPCC handoff
Doug Smith (dougbtv) Back Monday July 28

Hardware Warning

This model will not fit on a single DGX B200, even at FP4. Minimum is a full B300 node, GB300 NVL72, or MI355X system. Multi-node B200 may work. Coordinate with mgoin on what hardware is available for testing.

If Things Go Sideways

  • Build fails: Check CUDA version mismatch first (most common). If 3+ failures from merge artifacts, abandon the branch and start fresh with a new -v2 suffix.
  • Model won't load: Start with --enforce-eager --max-model-len 4096. Check if AttnRes or LatentMoE support landed correctly.
  • No SME help available: Self-serve from the upstream PR description, upstream Docker image (if available), and model README.
  • Dev box problems: Always have a fallback machine identified. Check chg status before committing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment