Automation: OpenEnv Examples Automation ID: openenv-examples Automation memory: $CODEX_HOME/automations/openenv-examples/memory.md
Run and expand on the examples to test and improve them.
Take the most recent three examples from the examples directory that have not been run based on memory and run them on a real-world workload.
TASK:
- Select three tasks, a mixture of training and inference, that you haven't run recently.
- Inference tasks can use Hugging Face inference providers.
- Training tasks can use Hug and Face jobs.
- Run the tasks on the remote infrastructure, monitor and review them. If you find any bugs, open separate bug-related pull requests for the example or openenv-core depending on where the bug is.
- For the example run itself do not open pull requests. Just return the experiment report in the chat.
- Try to make the experiments as mature as possible. Use evals, monitor metrics, create tables of results.
Use these libraries and resources where relevant:
- Harbor: https://www.harborframework.com/
- TRL: https://huggingface.co/docs/trl/en/index
- OpenEnv: https://huggingface.co/docs/openenv/index
- Inspect: https://inspect.aisi.org.uk/
Maintain memory across runs in the workspace: keep a structured experiment log with hypotheses, model/dataset/library versions, training configs, SLURM job ids, benchmark results, failures, and next harder experiments. Before starting new work, read the existing experiment log and avoid repeating completed or failed experiments unless there is a clear documented reason. Each run should progress toward harder problems or better controlled comparisons.
Cluster requirements:
- Use SLURM on hpc-cluster-hopper-login-node-1.
- Run 2-4 training jobs when appropriate.
- Use between 1 and 4 complete nodes.
- Jobs greater than 1 node must run on hopper-extra.
- Always use low or normal priority and respect cluster regulations. Do not use high priority for routine automation work, even if example commands show high priority.
- Useful commands include: squeue --me for job inspection; interactive debug sessions may use srun with suitable partition, node, GPU, time, CPU, and memory settings, but adapt qos to low or normal priority.
Bug tracking: when open source library bugs or sharp edges are found, do not open pull requests automatically. Maintain a table of issues with library, version/commit, reproduction steps, impact, workaround, and suggested patch so the user can later ask for PRs.
Expected output for each run: a concise status report covering new research findings, experiments launched or skipped with reasons, SLURM jobs and resource usage, benchmark/evaluation results, artifacts released or pending, Trackio dashboard links if available, newly discovered bugs, updates to the experiment log, and concrete next steps.
OUTPUT The output of this automation should be three reports, one on each example experiment, and potentially bug-related pull requests on openenv-core or the example.
PR Descriptions
- Extremely concise pull request descriptions, no validation, no testing plan, just a very limited description of what happens in one or two sentences.