Agentic Nightly Build — Phase 1 Demo Handout

agentic-nightly-handout.md

Agentic Nightly Build — Phase 1 Demo

JIRA: INFERENG-6206 Branches: nm-cicd | rhaiis-midstream-snippets

What is this?

An autonomous AI agent that drives the upstream vLLM → nm-vllm-ent midstream sync (merge, build, troubleshoot, report) without human intervention. The agent runs as a Kubernetes Job on the same cluster as our vLLM inference servers.

How it works

┌─────────────────────────────────────────────────────────────┐
│  K8s Job (IBM MI300X cluster, namespace: dtrifiro)          │
│                                                             │
│  ┌───────────────────────┐     ┌──────────────────────────┐ │
│  │  aicheat agent        │───▶│  Qwen3.5-397B (vLLM)     │ │
│  │  container            │     │  397B MoE, 262K context  │ │
│  │  - gh CLI + git       │◀───│  tool-call parsing       │ │
│  │  - skill instructions │     └──────────────────────────┘ │
│  │  - headless execution │                                  │
│  └───────────┬───────────┘                                  │
│              │                                              │
│              ▼                                              │
│  Outputs: pushed branches, build triggers, reports          │
└─────────────────────────────────────────────────────────────┘

• Agent harness: aicheat — a Python CLI that talks to OpenAI-compatible APIs (tool calling, file I/O, shell execution)
• Agent brain: Qwen/Qwen3.5-397B-A17B-FP8 served by vLLM on the MI300X cluster
• Container: quay.io/dosmith/aicheat-agent — Python 3.12 + aicheat + gh/git/curl/rg, with a build skill baked in
• Execution: Headless — no human in the loop. Prompt goes in via env var, stdin is closed, agent runs tool calls until done, exits cleanly

Phase 1 result: agent autonomously synced upstream v0.21.0

We gave the agent a simple task: merge upstream vLLM v0.21.0 into nm-vllm-ent and push the branch. Here's what it did on its own:

1. Generated a unique branch name (nightly/2026-05-15-184531)
2. Cloned nm-vllm-ent, added upstream remote, fetched the tag
3. Hit merge conflicts (.github/ modify/delete, a deleted requirements file, a deleted test) — resolved them correctly
4. Ran a safety check to verify the branch name before pushing
5. Hit a git credential error — self-recovered by configuring gh auth git-credential helper
6. Pushed the branch successfully (1158 commits ahead of main)
7. Printed a structured report and stopped

The agent was not told how to fix the credential issue — it figured that out by inspecting gh auth status and iterating. That's the kind of autonomous troubleshooting we're building toward for build failures.

Branch pushed by the agent: neuralmagic/nm-vllm-ent@nightly/2026-05-15-184531

The skill file

The agent gets a skill file baked into the container that covers:

• Safety guardrails: only push to nightly/ branches, max 3 build retry attempts
• Full workflow: merge → push → trigger build → monitor → assess failures → fix → retry → report
• Common fix patterns: pointers to Dockerfile.ubi (nm-vllm-ent) and neuralmagic/requirements/cuda.txt (nm-cicd) for the types of issues that come up (flashinfer pins, deep_gemm versions, missing packages, etc.)

What's next

• Phase 2: GHA workflows (build-agent-image.yml + agentic-nightly.yml) — trigger agent runs via gh workflow run instead of manual kubectl create
• Phase 3: Cron nightly schedule, Slack notifications, full deploy + smoke test loop
• Phase 4: Multi-model testing, PR validation, trend tracking

Quick reference

# Build the agent container locally
cd nm-cicd && ./agentic/build.sh

# Push to quay
PUSH=true ./agentic/build.sh

# Launch on cluster
KUBECONFIG=/tmp/kubeconfig-ocp-mi300x kubectl create -n dtrifiro -f charts/jobs/agentic-nightly.yml

# Watch it work
KUBECONFIG=/tmp/kubeconfig-ocp-mi300x kubectl logs -f -n dtrifiro -l app=agentic-nightly --tail=100