Prime RL - SciKnowEval Run Guide

SciKnowEval Run Guide.md

SciKnowEval RL — Beginner Run Guide (default prime-rl, GRPO only)

A step-by-step guide to reproduce cold-start GRPO training on SciKnowEval with prime-rl. By the end you will have RL-trained Qwen3-8B from scratch (no SFT warmstart) on multi-domain scientific MCQ and evaluated the result.

This guide is written from an actual run (Aug 2026) on an 8×H100 node. Everything here was verified to work. Companion to the reverse-text run guide — same tool, different task+model. Full repo (code, devlogs, plots): https://github.com/RohanAwhad/isdpo_reprod

"Default prime-rl" scope: this guide only needs vanilla, unmodified prime-rl plus one new (small, additive) environment package — no core prime-rl code changes. It reproduces the GRPO baseline only. The I-SDPO algorithm (this project's actual research contribution) needs extra code patches — see the repo above (prime_rl_patch/, devlogs.md) for that. GRPO doesn't need any of it: it's already exactly what orchestrator/algo/grpo.py ships with.

---

0. What are we actually doing?

The task: answer multiple-choice science questions (biology, chemistry, material science, physics) with just the correct letter.

Question: In the context of anaerobic metabolism, which of the following
correctly describes the function of S- or Se-methyltransferases...?
A. They regulate the transport of thiol and selenol metabolites
B. They (de)methylate thiol and selenol metabolites
C. They break down thiol and selenol metabolites
D. They catalyze the formation of thiol and selenol metabolites

Model output: B
Reward: 1.0 (exact match) or 0.0

Unlike the reverse-text guide, there is no SFT warmstart stage here — we start from the raw base-pretrained-with-chat-template checkpoint (PrimeIntellect/Qwen3-8B) and go straight to RL ("cold start").

Key concepts (60-second primer)

• prime-rl: an async RL training framework. One command (rl) spins up cooperating processes: an inference server (vLLM, generates completions), the orchestrator (runs the environment, scores rollouts), and the trainer (computes gradients, updates weights).
• sciknoweval environment: a new (not built into prime-rl) verifiers taskset — see step 4. Reward = 1.0 if the model's answer letter matches the gold answerKey, else 0.0. 800 questions (200/domain) are held out for eval, never seen in training.
• GRPO: samples group_size (16) completions per question, advantage = reward − group mean. This is prime-rl's default RL algorithm — nothing to install or patch for it.

---

1. Prerequisites

Hardware

• 1 Linux machine with ≥4 NVIDIA GPUs (H100/A100/H200 recommended; ≥8 to match this guide's exact config). Qwen3-8B needs real multi-GPU FSDP sharding for the trainer — unlike the 0.6B reverse-text guide's 1 GPU, budget ≥2 trainer GPUs (peak ~67 GB/GPU observed with a 2-way split) plus ≥2 inference GPUs.
• CUDA 12.8+ / driver 535+.
• ~200 GB free disk. The distributed checkpoint alone is ~98 GB for an 8B model (two ~49 GB shards); the HF-format weights export is another ~16 GB. Budget accordingly if you keep more than one checkpoint.

Accounts / keys

• Weights & Biases account + API key (https://wandb.ai/authorize). Verify it's yours before launching:

  curl -s https://api.wandb.ai/api/viewer -H "Authorization: $WANDB_API_KEY" | grep username

• The GPUs must be free. Check with nvidia-smi. On a shared node, also check no stray process is bound to the ports this guide uses (ss -ltnp | grep -E "29501|8000|8100|5555|5556|5000") — see the gotchas section for why this matters more than it sounds.

Software

• uv: curl -LsSf https://astral.sh/uv/install.sh | sh
• rsync, ssh, tmux.
• HF_TOKEN recommended (avoids rate limits downloading the 8B model / the SciKnowEval dataset from the Hub).

---

2. Get the repo onto the GPU node

Same as the reverse-text guide:

git clone https://github.com/PrimeIntellect-ai/prime-rl.git

# Sync to the GPU node (replace <gpu-node> with your host alias)
rsync -az \
  --exclude='.git' --exclude='.venv' --exclude='outputs' --exclude='logs' \
  --exclude='__pycache__' --exclude='.pytest_cache' --exclude='.ruff_cache' \
  prime-rl/ <gpu-node>:~/prime-rl/

Then ssh in:

ssh <gpu-node>
cd ~/prime-rl

All commands below run on the GPU node, inside ~/prime-rl, unless noted.

---

3. Set up the Python environment

uv sync --all-packages --extra flash-attn --extra disagg

• --extra flash-attn — same as the reverse-text guide (this node only has FA2, not FA3; required at import time).
• --extra disagg — new gotcha, not in the reverse-text guide: the rl launcher always shells out to a vllm-router binary to front the inference server, even for a simple single-node co-located setup with no actual prefill/decode disaggregation. That binary ships under the disagg extra. Without it:

  FileNotFoundError: [Errno 2] No such file or directory: 'vllm-router'
  

⚠️ Never run a bare uv sync or uv sync --all-packages with no --extra flags once the venv already has these installed — it will silently uninstall flash-attn and vllm-router (both are optional extras; a no-extras sync re-resolves to the default set and removes them), breaking every algorithm on the shared venv, not just this one. Always re-specify every extra you need on every sync.

---

4. Install the sciknoweval environment package

Unlike reverse-text's wheel-unzip workaround, this is a plain new directory — prime-rl auto-discovers any package under deps/prime-envs/environments/*/* as a uv workspace member, no pyproject.toml edits needed:

mkdir -p deps/prime-envs/environments/science/sciknoweval/sciknoweval

Copy in 3 files (from prime_rl_patch/sciknoweval_env/ in the repo above):

• pyproject.toml → deps/prime-envs/environments/science/sciknoweval/pyproject.toml
• sciknoweval/__init__.py, sciknoweval/mcq.py, sciknoweval/taskset.py → deps/prime-envs/environments/science/sciknoweval/sciknoweval/

Then re-sync (this both registers the new package and must re-include your extras, per the warning above):

uv sync --all-packages --extra flash-attn --extra disagg

You should see + sciknoweval==0.1.0 in the sync output.

Verify the install

uv run --no-sync python3 -c "
import sciknoweval, flash_attn
from sciknoweval.taskset import SciKnowEvalConfig, SciKnowEvalTaskset
import verifiers.v1 as vf
tasks = SciKnowEvalTaskset(SciKnowEvalConfig(task=vf.TaskConfig())).load()
print('OK, train tasks:', len(tasks))   # expect 17870
"
uv run --no-sync rl --help | head

As with reverse-text: always use uv run --no-sync for the rl command itself (plain uv run re-syncs against the lockfile, which doesn't know about your manually-added workspace member's install state the same way — safest to always pass --no-sync once you've synced deliberately).

---

5. Configure W&B

Same as the reverse-text guide — export WANDB_API_KEY=<your-key> and verify it resolves to your account.

---

6. Pre-download the model (optional but recommended)

PrimeIntellect/Qwen3-8B (a clone of Qwen/Qwen3-8B-Base with a multi-turn/tool-call chat template — the same "base, not instruction-tuned" relationship as PrimeIntellect/Qwen3-0.6B, so "cold start" means the same thing here as in the reverse-text guide) is ~16 GB:

uv run --no-sync python3 -c "
from huggingface_hub import snapshot_download
print(snapshot_download('PrimeIntellect/Qwen3-8B'))
"

The SciKnowEval dataset (hicai-zju/SciKnowEval, ~28k rows) downloads automatically the first time the taskset loads (a few seconds).

---

7. The config file

Save as examples/basic/sciknoweval/grpo.toml:

max_steps = 400
seq_len = 4096

[deployment]
num_train_gpus = 2
num_infer_gpus = 6

[model]
name = "PrimeIntellect/Qwen3-8B"

[trainer.model]
attn = "flash_attention_2"        # this node has FA2, not FA3

[monitors.wandb]
project = "sciknoweval-scratch"
name = "qwen3-8b-grpo-scratch"

[orchestrator]
batch_size = 128
group_size = 16                   # 8 questions x 16 rollouts per step

# Cold-start safety net: if the un-tuned base model doesn't reliably
# produce a parseable A-D letter, every rollout in a group scores exactly
# 0 and the group is degenerate -- monitor instead of enforcing so the run
# continues and shows the stall rather than hard-aborting after 10 empty
# batches. (In practice this didn't fire -- see step 9.)
[[orchestrator.post_batch_filters]]
type = "gibberish"

[[orchestrator.post_batch_filters]]
type = "repetition"

[[orchestrator.post_batch_filters]]
type = "zero_advantage"
enforce = false

[orchestrator.train.sampling]
max_completion_tokens = 256

[[orchestrator.train.source]]
name = "sciknoweval"

[orchestrator.train.source.env.taskset]
id = "sciknoweval"                # the env installed in step 4

[orchestrator.train.source.env.agent.harness]
id = "null"

[orchestrator.train.source.env.agent.runtime]
type = "subprocess"

[trainer.optim]
lr = 5e-6

[ckpt]                             # checkpoint at the end of training

[inference]

[inference.vllm]
data_parallel_size = 6             # matches num_infer_gpus

[orchestrator.renderer]
name = "prime-qwen3"

The two fields you'd change for a different model/scale: model.name and the [deployment] GPU split (keep num_train_gpus ≥2 for an 8B+ model; inference.vllm.data_parallel_size must equal num_infer_gpus).

---

8. Launch RL training

export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7   # all 8, matching 2+6 above

# max_steps=30 here (not the config's 400) so the run finishes *naturally*
# -- see the checkpoint-timeout gotcha below for why that matters.
tmux new-session -d -s sciknoweval-grpo \
  "cd ~/prime-rl \
   && export WANDB_API_KEY=$WANDB_API_KEY \
   && timeout --kill-after=60 1500 uv run --no-sync rl \
        @ examples/basic/sciknoweval/grpo.toml \
        --max-steps 30 \
        --output-dir outputs/sciknoweval-grpo \
        --run.name grpo-cold-start \
        --monitors.wandb.project sciknoweval-scratch \
        --monitors.wandb.name qwen3-8b-grpo-run1"

⚠️ Gotcha: don't cut the timeout close to the training time

Unlike the reverse-text guide's 10-min cap, set the timeout generously above your expected training time (here: 1500s for an expected ~13 min of training). Reason: [ckpt]'s end-of-training checkpoint write (distributed checkpoint + HF-format weight export) took ~10 more minutes on this node after the last training step — if timeout fires mid-write, you get a corrupted, unloadable checkpoint (shard files present but missing the .metadata file PyTorch's distributed checkpoint format needs) with no error until you try to load it later. Prefer --max-steps N set to something you expect to finish comfortably within your timeout, over relying on an external kill to stop a much-longer configured run early.

Watch it run

tmux attach -t sciknoweval-grpo   # Ctrl+B then D to detach
tail -f outputs/sciknoweval-grpo/grpo-cold-start/logs/latest/orchestrator.log
tail -f outputs/sciknoweval-grpo/grpo-cold-start/logs/latest/trainer.log

You'll see orchestrator lines like:

17:47:01 SUCCESS Step 1  |  20.4s | Reward 0.3438 | Trainable 64/128 (50.0%) | ...
17:58:17 SUCCESS Step 30 |  13.8s | Reward 0.7031 | ...
17:58:50 INFO    Writing final checkpoint
18:08:31 SUCCESS Training finished!

Expected timing (this run, 30 steps, 8×H100)

• Setup + inference pool boot: ~1–2 min
• 30 training steps: ~12.5 min (~15–25s/step once warm)
• Checkpoint + HF weight export: ~10 min — this is the surprising part relative to the reverse-text guide's 0.6B model; budget for it.
• Total wall-clock: ~23–25 min.

---

9. Evaluate the trained checkpoint

The run produces an HF-format model at outputs/sciknoweval-grpo/grpo-cold-start/weights/step_30/ (loadable by vLLM directly).

9a. Serve it

tmux new-session -d -s sciknoweval-eval \
  "cd ~/prime-rl && CUDA_VISIBLE_DEVICES=0 uv run --no-sync vllm serve \
     outputs/sciknoweval-grpo/grpo-cold-start/weights/step_30 \
     --served-model-name grpo-step30 --port 18000 \
     --gpu-memory-utilization 0.85 --max-model-len 4096"
until curl -s http://localhost:18000/v1/models >/dev/null; do sleep 5; done

(Port 18000, not 8000/9000 — both were already in use by other services on this shared node; check ss -ltnp and pick a free one.)

9b. Run the eval

eval_sciknoweval.py (in the repo above) is a small standalone script: loads the same held-out split="eval" 800 questions the taskset carves out, sends chat completions to the vLLM server, extracts the answer letter, reports accuracy. Copy it to the prime-rl repo root on the GPU node (scp eval_sciknoweval.py <gpu-node>:~/prime-rl/), then:

uv run --no-sync python3 eval_sciknoweval.py \
  --base-url http://localhost:18000/v1 --model grpo-step30 \
  --n-samples 1 --concurrency 32 --out /tmp/eval-grpo.json

Expected result (this run, 30 steps)

{
  "overall_accuracy": 0.7425,
  "per_domain_accuracy": {"Biology": 0.79, "Chemistry": 0.725, "Material": 0.69, "Physics": 0.765}
}

Compare to the pre-RL base model (serve PrimeIntellect/Qwen3-8B directly instead of the checkpoint dir, same eval script): 53.0%. 30 steps of cold-start GRPO drove +21.25 points.

9c. Clean up

tmux kill-session -t sciknoweval-eval
tmux kill-session -t sciknoweval-grpo   # if still around

---

10. Gotchas & lessons (read these — they cost real time)

1. uv sync/uv sync --all-packages with no --extra flags silently uninstalls flash-attn and vllm-router if the venv's current state came from a sync that included them. Always re-specify every extra.
2. vllm-router isn't in the default deps — it's under --extra disagg, even though the launcher needs it for a simple single-node setup (not just real disaggregated serving). Missing it fails with FileNotFoundError: 'vllm-router'.
3. End-of-training checkpoint writes can take ~10 minutes for an 8B model on a contended shared filesystem — budget your timeout generously above expected training time, or you'll get a checkpoint with full-size shard files but no .metadata (silently unloadable until you actually try to resume from it).
4. Processes stuck writing a checkpoint can enter Linux D state (uninterruptible sleep on disk I/O) and cannot be killed with kill -9 until the underlying I/O completes — not even by the parent timeout's SIGKILL. If you see GPU memory stuck "in use" with no compute process listed in nvidia-smi, check ps -o stat for D state; there's no workaround except waiting (or using different GPUs for your next job in the meantime).
5. Fixed ports can collide on a shared node. The weight-broadcast NCCL rendezvous defaults to port 29501; two back-to-back runs where the first's port isn't released yet fail with Address already in use — but the orchestrator doesn't always propagate this as a fatal error, so you can burn your whole timeout budget on a run that produced zero training steps. Override with --trainer.weight-broadcast.type nccl --trainer.weight-broadcast.port <N> --orchestrator.weight-broadcast.type nccl --orchestrator.weight-broadcast.port <N> (both sides, both flags — the type must be set explicitly alongside port or the CLI resolves a different discriminated-union variant). Also check ss -ltnp for ports 9000/8000 etc. before picking an eval server port — other unrelated services on a shared node commonly squat on the obvious defaults.
6. --deployment.num-train-gpus/num-infer-gpus (not a CUDA_VISIBLE_DEVICES-only split) control the trainer/inference GPU allocation; inference.vllm.data_parallel_size must match num-infer-gpus or the inference pool won't use all the GPUs you gave it.
7. The base PrimeIntellect/Qwen3-8B clone is already substantially better than chance cold (53% vs. a naive 25% 4-way-random-guess baseline) — unlike reverse-text's total cold-start collapse, GRPO's "all rollouts tied at reward 0" degenerate case is rare here, not the default state. Don't be surprised if the zero_advantage monitor filter basically never fires.

---

11. File map

outputs/sciknoweval-grpo/grpo-cold-start/
├── configs/            # resolved subconfigs written by the launcher
├── logs/latest/
│   ├── orchestrator.log   # per-step reward, routing/trainable fraction
│   └── trainer.log        # per-step loss/entropy/grad-norm/peak-mem
├── checkpoints/step_30/trainer/   # distributed checkpoint (.distcp + .metadata)
├── weights/step_30/    # ← HF-format model, loadable by vLLM (use this for eval)
├── rollouts/step_*/     # saved rollouts per step
└── metrics.jsonl        # one JSON record per orchestrator step

---

12. Quick reference (copy-paste to re-run end-to-end)

# On the GPU node, in ~/prime-rl (after steps 3-4 setup once)
export WANDB_API_KEY=<your-key>
export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7

# Train (generous timeout — see gotcha #3)
tmux new-session -d -s sciknoweval-grpo \
  "cd ~/prime-rl && timeout --kill-after=60 1500 uv run --no-sync rl \
     @ examples/basic/sciknoweval/grpo.toml --max-steps 30 \
     --output-dir outputs/sciknoweval-grpo --run.name grpo-cold-start \
     --monitors.wandb.project sciknoweval-scratch --monitors.wandb.name qwen3-8b-grpo-run1"

# Wait ~25 min, then eval:
tmux new-session -d -s sciknoweval-eval \
  "cd ~/prime-rl && CUDA_VISIBLE_DEVICES=0 uv run --no-sync vllm serve \
     outputs/sciknoweval-grpo/grpo-cold-start/weights/step_30 \
     --served-model-name grpo-step30 --port 18000 --gpu-memory-utilization 0.85 --max-model-len 4096"
until curl -s http://localhost:18000/v1/models >/dev/null; do sleep 5; done
uv run --no-sync python3 eval_sciknoweval.py \
  --base-url http://localhost:18000/v1 --model grpo-step30 --n-samples 1 --out /tmp/eval-grpo.json

# Clean up
tmux kill-session -t sciknoweval-eval; tmux kill-session -t sciknoweval-grpo

---

Verified run: 8× H100 node, 30 steps, ~25 min total (12.5 min training + ~10 min checkpoint write). Pre-RL base accuracy 53.0% → post-RL eval accuracy 74.25% (mean@1, 800 held-out MCQ).