Graham grahama1970

Review Bundle Findings — LiteLLM Readiness Smokes

1. Regression Snapshot

Command: . .venv/bin/activate && READINESS_LIVE=1 STRICT_READY=1 READINESS_EXPECT=all_smokes_core python scripts/mvp_check.py
Failure: tests/smoke_optional/test_agent_proxy_validation_errors.py::test_agent_proxy_headers_precedence
Root symptom: headers recorded by the tool invoker are {} instead of request overrides ("X-Env": "B", "X-Req": "C").

2. Primary Root Cause

tests/smoke_optional/test_agent_proxy_headers_and_errors.py replaces litellm/experimental_mcp_client/mini_agent/agent_proxy.HttpToolsInvoker with _FakeInvoker and never restores it. Because pytest runs smoke_optional tests in a single process, the override survives into the next test and prevents the header precedence contract from being exercised with the real invoker.
Evidence:

Codex Review Bundle (Single File)

Repo root: /home/graham/workspace/experiments/codex
Date: 2025-09-28

Executive Verdict

Readiness: ✅ Top risks are low and localized to config and CLI glue.

Findings by Focus Area

External Review Prompt (Canonical v3)

Goal: Deliver a blunt, evidence‑backed production‑readiness assessment and a minimal patch set (unified diffs) with tests and doc updates. No broad refactors. Ship safety first.

Reviewer persona & tone

Principal SRE/DevOps + AppSec mindset; fluent with Linux userland (systemd, Docker/OCI, rsync/tar), CI, and Python/Node toolchains.
Be terse, specific, and fail‑closed. Unverified claims must be called out and marked 🔴 or 🟡.

Project context (declare at top of your report)

Project:

Codex Review Bundle (Single File)

Repo root: /home/graham/workspace/experiments/codex
Date: 2025-09-28

Executive Verdict

Readiness: ✅ Top risks are low and localized to config mapping and CLI glue.

Findings by Focus Area

Codex Review Bundle Prompt (Project‑Tailored)

Goal

Produce a concise, evidence‑backed technical review of Codex (Rust workspace: CLI/TUI/core), with minimal patches (unified diffs), tests, and docs. Return a single public Gist URL that contains the full review bundle.

Reviewer profile & tone

Senior Rust engineer with AppSec + CLI/TUI experience (cargo, clippy, ratatui, insta snapshots, MCP). Terse, specific, fail‑closed. Mark unverified claims 🔴/🟡.

Project anchors

Repo root:

All‑Smokes Gate Still Timing Out/Fails in Split — Targeted Debug + Patch Requests Created: 2025-09-27 TTL: Private, delete within 15 minutes after review

Summary

We split the composite all_smokes gate into all_smokes_core + all_smokes_nd and added per‑check timeouts, xdist and PYTEST_ADDOPTS pass‑through.
The orchestrator no longer dies universally, but we still see:
1. Harness timeouts under certain runs (improved but still possible on slow hosts).
2. True FAILs in all_smokes_nd due to env/base mismatches (see below) — these are not red in isolation.

Mini‑Agent + Router Readiness — All‑Smokes Orchestration Timeout (Request for Focused Help) Created: 2025-09-27 TTL: Please treat as private and ephemeral; delete within 15 minutes after review.

Context and Goal

Project: LiteLLM fork with env‑gated codex‑agent provider and Mini‑Agent + Readiness system.
Deploy gate we want: make project-ready-all must pass, i.e., EVERY smoke (smoke, smoke_optional, ndsmoke, ndsmoke_e2e) green.
Current state: All failing clusters are fixed; individual smokes pass in isolation. The composite readiness check all_smokes times out in the harness despite extending per‑check timeout.

What’s Working (evidence-based)

CodeWorld — External Review Questions, Context, and Code Anchors

Context: CodeWorld is a prompt‑driven, multi‑variant orchestrator for agentic code generation. It emits per‑instance prompts, autostarts a tiny FastAPI ingest backend, runs agents (or a local fallback), and aggregates a reproducible scorecard. Observability flows to ArangoDB with a thin proto dashboard. Memory hooks integrate a Graph Memory service for recall and timeline context.

Inspiration: CWM: An Open‑Weights LLM for Research on Code Generation with World Models (Meta AI, Sept 24, 2025). Local copy: docs/papers/CWM_ An Open-Weights LLM for Research on Code Generation with World Models _ Research - AI at Meta.md. Our aim is to explore world‑model style signals for agentic coding by capturing observation→action episodes during runs and enabling recall‑driven guidance.

Objective: Harden the orchestrator for research‑grade iteration while keeping it thin and deterministic by default. We want principled process lifecycle, secure defaults,

	# Source: litellm/experimental_mcp_client/mini_agent/agent_proxy.py (lines 160-239)
	if bool(inv.get("ok")):
	return inv
	return None
	except Exception:
	return None

	# Compute escalation metadata eagerly
	escalated = bool(req.escalate_on_budget_exceeded and req.escalate_model)
	used_model = req.escalate_model if escalated else req.model

	# Project Bundle

	- Generated: 2025-09-27T17:43:50Z
	- Root: /home/graham/workspace/experiments/codex
	- Git: 5c67dc3+dirty
	- Files: 205
	- Bundle Part: 1
	- Context Tokens Limit: 400000

	---