Automation: Post-training agent experiments Automation ID: post-training-agent-experiments Automation memory: $CODEX_HOME/automations/post-training-agent-experiments/memory.md Last run: 2026-06-23T05:01:48.919Z (1782190908919)
Run the Training Agents Terminal-Bench loop.
GOAL Train an approximately 2B parameter open model to exceed 40 on Terminal-Bench. If full Terminal-Bench scoring is blocked, use the strongest available proxy gate, such as TB Lite, Harbor, Inspect, or an OpenEnv tbench2-style held-out subset, but label it as a proxy and do not claim a full benchmark win.
Loop shape: GOAL -> DISCOVER -> PLAN -> EXECUTE -> VERIFY -> ITERATE. This is a stateful loop, not an isolated run. Start every run by reading the existing automation memory and repo logs, then continue from the latest recorded loop state.
Before doing new research or launching jobs, read:
- /Users/ben/.codex/automations/post-training-agent-experiments/memory.md
- research/notes.md
- research/results.tsv
- research/issues.md or any repo-local bug table that exists
- docs/terminal-bench-loop.md if present
If memory still shows an active or unresolved run, inspect it first. As of 2026-06-23, the latest recorded active work was the Gemma 4 E2B terminal GRPO sweep: inspect SLURM jobs 22298242 and 22298248, parse any TBLite eval summaries under /fsx/benjamin_burtenshaw/post-training-agent-experiments/2026-06-23-gemma4-terminal-sweep/evals/, update research/results.tsv, and publish only if an adapter beats base on the proxy benchmark.
DISCOVER Research current techniques and sources before choosing the next rung. Prefer Hugging Face Papers, official TRL docs/examples, OpenEnv docs/examples, Harbor docs, Inspect docs, source repos, and current Hub datasets or models. Avoid excessively old or deprecated methods such as PPO unless there is a specific modern comparison reason. Focus on TRL-native SFT, GRPO, DAPO or Dr. GRPO variants, environment GRPO, self-distillation, verifier rewards, and terminal-agent data.
Use these resources where relevant:
- Harbor: https://www.harborframework.com/
- TRL: https://huggingface.co/docs/trl/en/index
- OpenEnv: https://huggingface.co/docs/openenv/index
- Inspect: https://inspect.aisi.org.uk/
Do not train on Terminal-Bench eval tasks, TB Lite eval tasks, public leaderboard solutions, official eval trajectories, or hidden labels. Eval-only sources may be used for schema inspection and scoring.
PLAN Design one experiment or sweep that advances the recorded loop state. Record the planned rung before launching jobs:
- model and tokenizer
- method: SFT, GRPO, environment GRPO, self-distillation, or model comparison
- dataset and train/eval split policy
- reward or eval definition
- smallest smoke command
- sweep size, seeds, learning rates, and node count
- Trackio project or Space
- artifact destination
- score gate and next decision
Use repo-local skills when the task matches them:
- TRL Post-Training for method/trainer shape
- TRL SFT for SFT warm starts or trace imitation
- OpenEnv Agentic RL for environment contracts and rollouts
- Agentic Self-Distillation for verified trace recycling
- Trackio Observability for Trackio, metrics, logs, and artifacts
- Hugging Face CLI Workflows for HF CLI, Hub, Jobs, buckets, and persistence
Use project sub-agents when parallel work is useful and available: research-scout for current sources, training-planner for the method queue, trl-implementer for one coherent implementation, script-runner for commands and failure summaries, tracking-reporter for SLURM/Trackio/artifacts, openenv-builder for environment contracts, self-distillation-designer for trace loops, and integrity-reviewer for leakage/eval/release claims. Keep write-heavy work to one implementation path at a time.
EXECUTE Run only work that advances the loop. Prefer small smoke tests before full sweeps. Run 2-4 training jobs when capacity and project state allow.
Cluster requirements:
- Use SLURM on hpc-cluster-hopper-login-node-1.
- Use between 1 and 4 complete nodes.
- Jobs greater than 1 node must run on hopper-extra.
- Always use low or normal priority and respect cluster regulations. Do not use high priority for routine automation work.
- Useful commands include /opt/slurm/bin/squeue --me and sacct for job inspection; interactive debug sessions may use srun with suitable partition, node, GPU, time, CPU, and memory settings, with qos low or normal.
- Prefer FSx-backed caches, virtualenvs, run roots, and temporary directories.
Every remote run should have an exact launch command or SLURM script, job id, model/dataset/library versions, Trackio dashboard for long or remote training, persistent artifact path, and checkpoint save before any in-process post-eval that might hang.
VERIFY Evaluate checkpoints with a verifier separate from training whenever possible. Verification order:
- smoke test for tokenization, reward parsing, logging, and checkpoint save
- held-out task or proxy gate, such as TB Lite or OpenThoughts-style held-out tasks
- Harbor, Inspect, or OpenEnv tbench2 evaluation
- full Terminal-Bench score when infrastructure is available
Do not promote a checkpoint from training reward alone. Promote only when held-out evaluation improves over the base model or previous best adapter. The target success gate is Terminal-Bench score above 40; proxy improvements are useful loop evidence but not final success.
ITERATE If the score does not beat 40, feed the result back into the next run. Record what changed, what metric moved or failed, the failure mode, whether the bottleneck is model capacity/data/reward/environment/eval/infrastructure, and the next harder or better-controlled rung.
Do not repeat failed variants without a documented material change in method, reward, model, data, evaluator, or infrastructure. If a rung has no reward variance, redesign prompt/task grouping before scaling. If syntax improves but task success stays flat, move to semantic data or a stronger model comparison. If proxy score is still 0, avoid full Terminal-Bench claims.
Objective for successful releases: if an open checkpoint improves over base or previous best on the target benchmark/proxy and is a credible promotion candidate, release it to the Hugging Face Hub with a Trackio dashboard and a concise README report explaining what changed, what benchmark improved, and how to reproduce the result.
Bug tracking: when open source library bugs or sharp edges are found, do not open pull requests automatically. Maintain a table of issues with library, version/commit, reproduction steps, impact, workaround, and suggested patch so the user can later ask for PRs.
Expected output for each run: concise status report covering loop stage, score gate, new research findings, experiments launched or skipped with reasons, SLURM jobs and resource usage, benchmark/evaluation results, artifacts released or pending, Trackio dashboard links, newly discovered bugs, memory/log files updated, and concrete next loop state.