Reverse-Text RL — Beginner Run Guide

A step-by-step guide to reproduce the reverse-text RL experiment with prime-rl. By the end you will have fine-tuned Qwen3-0.6B with RL to reverse text, starting from a warm-started (SFT) checkpoint, and evaluated the result.

This guide is written from an actual run (Aug 2026) on an 8×H100 node. Everything here was verified to work.

0. What are we actually doing?

The task: teach a small language model to reverse a chunk of text character-by-character.

Input:  "The community in Bruck was merged into it"
Output: ".ti ot demerg saw kcuBr ni ytinummoC ehT"

The pipeline has two stages (we only run stage 2 — stage 1 is already done for us):

Stage	What	Model produced	Reward (reverse-text env)
1. SFT warmup	Supervised fine-tune on 1K reversed-Wikipedia paragraphs	`PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT`	~0.09
2. RL	Reinforcement learning on the `reverse-text` env	your checkpoint	~0.78

We warm-start RL from the SFT checkpoint (stage 1's output, already on HuggingFace), then run 20 RL steps. Reward climbs from ~0.09 → ~0.78.

Key concepts (60-second primer)

prime-rl: an async RL training framework. One command (rl) spins up 3 cooperating processes:
- Inference server (vLLM) — generates model completions. Needs 1 GPU.
- Orchestrator — runs the environment, scores rollouts, ships data around. CPU.
- Trainer — computes gradients and updates weights. Needs 1 GPU.
reverse-text environment: the task/scoring function. Reward = how close the model's output is to the true reversal (longest-common-substring based, 0–1).
warm-start checkpoint: the model weights we start RL from = PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT.
rollout: one sample = (prompt → model completion → reward). Batch = 128, 16 rollouts per example.

1. Prerequisites

Hardware

1 Linux machine with ≥2 NVIDIA GPUs (H100/A100/H200). Design is 1 trainer GPU + 1 inference GPU.
CUDA 12.8+ / driver 535+ (we used CUDA 13.3, driver 610).
~50 GB free disk (for the uv env + model cache).

Accounts / keys

Weights & Biases account + API key. Get it from https://wandb.ai/authorize.
⚠️ Verify the key is YOURS before launching. In this run, the machine's WANDB_API_KEY env var actually belonged to a teammate, so the run silently logged to their account. Check with:
```
curl -s https://api.wandb.ai/api/viewer -H "Authorization: $WANDB_API_KEY" | grep username
```
The GPUs must be free (no other processes on them). Check with nvidia-smi.

Software

uv (Python package manager): https://docs.astral.sh/uv/
```
curl -LsSf https://astral.sh/uv/install.sh | sh
```
rsync, ssh, tmux (for running on a remote node).
HuggingFace access is anonymous-friendly for these public models (no token strictly required, but set HF_TOKEN to avoid rate limits).

2. Get the repo onto the GPU node

If the repo isn't already on the node, copy it there. From your local machine:

# Clone if you don't have it locally
git clone https://github.com/PrimeIntellect-ai/prime-rl.git

# Sync to the GPU node (replace rh-h100-04 with your host alias)
rsync -az \
  --exclude='.git' --exclude='.venv' --exclude='outputs' --exclude='logs' \
  --exclude='__pycache__' --exclude='.pytest_cache' --exclude='.ruff_cache' \
  prime-rl/ <decided-node>:~/prime-rl/

Then ssh in:

ssh <one of the nodes>
cd ~/prime-rl

All commands below run on the GPU node, inside the repo dir (~/prime-rl), unless noted.

3. Set up the Python environment

prime-rl uses uv to manage a pinned environment from uv.lock.

3a. Install the base deps + flash-attn

uv sync --extra flash-attn

This installs torch (cu128), vLLM, transformers, wandb, flash-attn (prebuilt wheel), etc. Takes a few minutes (fast if uv cache is warm).

3b. Install the `reverse-text` environment package ⚠️ (gotcha)

The reverse-text task is an optional extra (envs). The "obvious" command fails:

# ❌ FAILS — the `envs` extra includes `mini-swe-agent-plus`, whose wheel 404s on the primeintellect hub
uv sync --extra envs

And uv pip install reverse-text also fails because uv's resolver chokes on reverse-text's torch build-dependency metadata.

Workaround — install the wheel manually by unzipping it into the venv (it's a pure-Python single module):

# Find the site-packages dir of the venv
SP=$(uv run python -c "import site; print(site.getsitepackages()[0])")

# Download the latest reverse-text wheel from the primeintellect index
curl -sL -o /tmp/reverse_text.whl \
  "https://hub.primeintellect.ai/primeintellect/reverse-text/@e3cd3fe4/reverse_text-0.1.5-py2.py3-none-any.whl"

# Unzip (install) it into the venv
unzip -oq /tmp/reverse_text.whl -d "$SP"

Why this works: reverse_text is a pure-Python package (one file, reverse_text.py). Its only real runtime deps (vLLM, verifiers, torch) are already in the venv from step 3a. Unzipping the wheel into site-packages is equivalent to pip install --no-deps.

3c. Verify the install

uv run --no-sync python -c "import reverse_text, flash_attn; print('OK')"
uv run --no-sync rl --help | head

Important — always use uv run --no-sync (not uv run) for the rl command. Plain uv run re-syncs the venv to match the lockfile, which removes the manually-installed reverse-text package. The --no-sync flag skips that check. (The subprocesses rl spawns — inference, orchestrator, torchrun — are venv scripts and don't re-sync, so they're fine.)

4. Configure W&B

# Make sure YOUR key is exported
export WANDB_API_KEY=<your-key>

# Sanity check it resolves to your account
curl -s https://api.wandb.ai/api/viewer -H "Authorization: $WANDB_API_KEY" | grep -o '"username":"[^"]*"'

The run config (examples/reverse_text/rl.toml) already sets the W&B project to reverse-text. You can override the run name on the command line.

5. Pre-download the warm-start model (optional but recommended)

Doing this before the timed run avoids burning your 10-minute budget on a model download (the 0.6B model is ~1.2 GB, ~25 s on a decent link):

uv run --no-sync python -c "
from huggingface_hub import snapshot_download
print(snapshot_download('PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT'))
"

6. The config file

Everything is driven by examples/reverse_text/rl.toml:

max_steps = 20          # total RL steps
seq_len = 2048

[model]
name = "PrimeIntellect/Qwen3-0.6B-Reverse-Text-SFT"   # ← the warm-start checkpoint

[wandb]
project = "reverse-text"
name = "reverse-text"

[orchestrator]
batch_size = 128            # 8 examples × 16 rollouts
rollouts_per_example = 16

[orchestrator.train.sampling]
max_completion_tokens = 128

[[orchestrator.train.env]]
id = "reverse-text"         # the env we installed in step 3b

[trainer.optim]
lr = 3e-6

[ckpt]                      # save a checkpoint at the end of training

[inference]                 # empty = use vLLM defaults (port 8000)

The key field: model.name is the warm-start checkpoint. You don't need to change it — it already points at the SFT model.

7. Launch RL training

We run inside a tmux session so the job survives an ssh disconnect, and wrap it in timeout to enforce a hard time cap.

# Pick 2 free GPUs (here: 0 and 1). Verify with nvidia-smi first.
export CUDA_VISIBLE_DEVICES=0,1

# 600 s = 10 min hard cap. --kill-after=30 sends SIGKILL if cleanup stalls.
tmux new-session -d -s reverse-text-rl \
  "cd ~/prime-rl \
   && export WANDB_API_KEY=$WANDB_API_KEY \
   && CUDA_VISIBLE_DEVICES=0,1 timeout --kill-after=30 600 uv run --no-sync rl \
        @ examples/reverse_text/rl.toml \
        --output-dir outputs/reverse-text-rl \
        --wandb.project reverse-text \
        --wandb.name reverse-text-run1"

What this command does

rl @ examples/reverse_text/rl.toml — the unified launcher. It:
1. Writes subconfigs to outputs/reverse-text-rl/configs/.
2. Starts the inference server on GPU 0 (port 8000).
3. Starts the orchestrator (CPU).
4. Starts the trainer (via torchrun) on GPU 1.
--output-dir — where checkpoints, logs, rollouts, and W&B local files go.
--wandb.project / --wandb.name — override the W&B destination.
timeout 600 — kills the whole tree after 10 min (the launcher's SIGTERM handler cleans up child processes).

Watch it run

# Attach to the tmux session to see live trainer progress
tmux attach -t reverse-text-rl
# (Ctrl+B then D to detach without killing it)

# Or tail the per-component logs
tail -f outputs/reverse-text-rl/logs/trainer.log
tail -f outputs/reverse-text-rl/logs/orchestrator.log
tail -f outputs/reverse-text-rl/logs/inference.log

You'll see trainer steps like:

17:43:39 SUCCESS Step 0 | Time: 47.75s | Loss: 0.0034 | Entropy: 0.6677 | ...
17:44:43 SUCCESS Step 19 | ...
17:44:44 INFO Writing final checkpoint
17:45:51 SUCCESS RL training finished!

Expected timing (this run)

Setup + inference server boot: ~1 min
20 training steps: ~1 min 53 s (step 0 is slow ~48 s due to compile/warmup; the rest ~1.7–3.3 s each)
Checkpoint writing: ~1 min
Total wall-clock: ~2 min 53 s — far under the 10-min cap.

8. Evaluate the trained checkpoint

The run produces an HF-format model at outputs/reverse-text-rl/weights/step_20/ (loadable by vLLM directly) and a distributed checkpoint at outputs/reverse-text-rl/checkpoints/step_20/.

8a. Start an inference server with the trained model

tmux new-session -d -s rt-eval \
  "cd ~/prime-rl \
   && CUDA_VISIBLE_DEVICES=0 uv run --no-sync inference \
        --model.name outputs/reverse-text-rl/weights/step_20"

Wait for it to be ready (~1–2 min):

until curl -s http://localhost:8000/health >/dev/null; do sleep 5; done && echo "server ready"

8b. Run the eval

# Uses the `reverse-text` env, 20 examples × 3 rollouts, max 1024 tokens
uv run --no-sync vf-eval reverse-text \
  -m outputs/reverse-text-rl/weights/step_20 \
  -b http://localhost:8000/v1 \
  -n 20 --max-tokens 1024

Expected result

reward: avg - 0.784, std - 0.099
pass@1: 0.967    pass@2: 1.000

Compare to the pre-RL reward (step 0, the warm-started SFT model): 0.0907. RL drove the gain from ~0.09 → ~0.78.

8c. Clean up

tmux kill-session -t rt-eval
tmux kill-session -t reverse-text-rl   # if still around

9. The reward trajectory (from this run)

Step	Reward	Notes
0	0.0907	warm-start SFT ckpt, before any RL update
1	0.1039
5	0.3496
10	0.6590
15	0.7648
19	~0.75	last training step
eval	0.784	final vf-eval (20×3 rollouts)

The loss stayed near 0 throughout (the model was already close from SFT); the skill gain shows up in reward, not loss.

10. Gotchas & lessons (read these — they cost real time)

Wrong W&B key. The machine's WANDB_API_KEY env var may belong to someone else. Always verify it resolves to your account (step 4) before launching, or the run logs to a teammate's project silently.
--extra envs is broken. It pulls mini-swe-agent-plus, whose wheel 404s. Install reverse-text manually (step 3b).
uv pip install reverse-text fails on a torch build-dep resolver quirk. Unzip the wheel into site-packages instead (step 3b).
Always use uv run --no-sync for the rl launcher. Plain uv run re-syncs and wipes the manually-installed reverse-text. (Subprocesses spawned by rl are fine — they call the venv scripts directly.)
flash-attn is required, not optional. Without the --extra flash-attn install, the rl entrypoint fails at import time (ring_flash_attn → flash_attn).
Pin free GPUs with CUDA_VISIBLE_DEVICES. The launcher grabs GPU 0 (inference) and GPU 1 (trainer) by default. On a shared 8-GPU node, set CUDA_VISIBLE_DEVICES to 2 free IDs to avoid collisions.
No wall-clock time limit in config. prime-rl only has max_steps, not a max-time option. Enforce the 10-min cap externally with timeout 600.
Repo isn't on the node by default. rsync it (step 2). But the node likely has a big uv cache, so the heavy uv sync is fast.
reverse-text is a pure-Python single module (reverse_text.py) built on verifiers. It registers itself as an env via load_environment().

11. File map (where things land)

outputs/reverse-text-rl/
├── configs/            # resolved subconfigs written by the launcher
│   ├── inference.toml
│   ├── orchestrator.toml
│   └── trainer.toml
├── logs/
│   ├── inference.log   # vLLM server logs
│   ├── orchestrator.log# env + rollout logs, per-step reward
│   ├── trainer.log     # per-step loss/entropy/grad-norm
│   └── envs/train/reverse-text/env_server.log
├── checkpoints/step_20/trainer/   # distributed checkpoint (.distcp)
├── weights/step_20/    # ← HF-format model, loadable by vLLM (use this for eval)
├── rollouts/step_*/    # saved rollouts per step
└── wandb/              # local W&B run files

12. Quick reference (copy-paste to re-run end-to-end)

# On the GPU node, in ~/prime-rl
export WANDB_API_KEY=<your-key>
export CUDA_VISIBLE_DEVICES=0,1

# Train (10-min cap)
tmux new-session -d -s rt-train \
  "cd ~/prime-rl && CUDA_VISIBLE_DEVICES=0,1 \
   timeout --kill-after=30 600 uv run --no-sync rl \
     @ examples/reverse_text/rl.toml \
     --output-dir outputs/reverse-text-rl \
     --wandb.project reverse-text --wandb.name reverse-text-run1"

# After it finishes (~3 min), eval:
tmux new-session -d -s rt-eval \
  "cd ~/prime-rl && CUDA_VISIBLE_DEVICES=0 uv run --no-sync inference \
     --model.name outputs/reverse-text-rl/weights/step_20"
until curl -s http://localhost:8000/health >/dev/null; do sleep 5; done
uv run --no-sync vf-eval reverse-text \
  -m outputs/reverse-text-rl/weights/step_20 \
  -b http://localhost:8000/v1 -n 20 --max-tokens 1024

# Clean up
tmux kill-session -t rt-eval; tmux kill-session -t rt-train

Verified run: rh-h100-04, 2× H100, 20 steps, ~2m53s total. Pre-RL reward 0.0907 → post-RL eval reward 0.784.

RohanAwhad/Prime RL - Reverse Text Run Guide.md

Select an option

No results found

Select an option

No results found

Reverse-Text RL — Beginner Run Guide

0. What are we actually doing?

Key concepts (60-second primer)

1. Prerequisites

Hardware

Accounts / keys

Software

2. Get the repo onto the GPU node

3. Set up the Python environment

3a. Install the base deps + flash-attn

3b. Install the `reverse-text` environment package ⚠️ (gotcha)

3c. Verify the install

4. Configure W&B

5. Pre-download the warm-start model (optional but recommended)

6. The config file

7. Launch RL training

What this command does

Watch it run

Expected timing (this run)

8. Evaluate the trained checkpoint

8a. Start an inference server with the trained model

8b. Run the eval

Expected result

8c. Clean up

9. The reward trajectory (from this run)

10. Gotchas & lessons (read these — they cost real time)

11. File map (where things land)

12. Quick reference (copy-paste to re-run end-to-end)

RohanAwhad/Prime RL - Reverse Text Run Guide.md

Reverse-Text RL — Beginner Run Guide

0. What are we actually doing?

Key concepts (60-second primer)

1. Prerequisites

Hardware

Accounts / keys

Software

2. Get the repo onto the GPU node

3. Set up the Python environment

3a. Install the base deps + flash-attn

3b. Install the reverse-text environment package ⚠️ (gotcha)

3c. Verify the install

4. Configure W&B

5. Pre-download the warm-start model (optional but recommended)

6. The config file

7. Launch RL training

What this command does

Watch it run

Expected timing (this run)

8. Evaluate the trained checkpoint

8a. Start an inference server with the trained model

8b. Run the eval

Expected result

8c. Clean up

9. The reward trajectory (from this run)

10. Gotchas & lessons (read these — they cost real time)

11. File map (where things land)

12. Quick reference (copy-paste to re-run end-to-end)

3b. Install the `reverse-text` environment package ⚠️ (gotcha)