Skip to content

Instantly share code, notes, and snippets.

@donbr

donbr/session16-rlvr-cheatsheet.md

Created July 24, 2026 00:49
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save donbr/b3ee700491dd5f120adaf4c2326fe8a4 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save donbr/b3ee700491dd5f120adaf4c2326fe8a4 to your computer and use it in GitHub Desktop.
Download ZIP
Session 16 Cheat Sheet — RLVR (Reinforcement Learning with Verifiable Rewards)
Raw
session16-rlvr-cheatsheet.md

Session 16 Cheat Sheet — RLVR (Reinforcement Learning with Verifiable Rewards)

A frame to help you reason through the assignment — concepts, diagrams, and the API map. It deliberately does not contain the answers or filled-in activity code. Instead it gives you the questions to ask yourself and the method to get there. The work — and the learning — is in running the cells, reading the completions your verifier rejected, and writing your own conclusions.

Notebook: 01_RLVR_Verifiable_Rewards.ipynb (34 cells) · README: 16_RLVR/README.md No corpus and no GPU: the policy is an API model (gpt-4.1-nano) sampled at temperature 1.0.

1. Quick Reference

You want to… Reach for One-liner
Sample one completion from the policy simple_complete simple_complete(prompt, system="...", temperature=1.0)
Carry ground truth alongside a prompt Problem Problem("What is 12 * 13?", "156") → .question, .answer
Pull the final answer out of a completion extract_number first \boxed{...}, else the last number in the text, else ""
Make "80" match "80.0" MathRewardFunction._normalize float(value) when it parses, else value.strip()
Turn a check into a scalar reward MathRewardFunction.compute reward_fn.compute(response, problem.answer) → +1.0 / -0.1
Sample a group and verify every member sample_and_verify sample_and_verify(problem, n_samples=4) → list[Sample]
Flag a suspicious verified-correct sample looks_like_hack < 20 words or < 2 numbers outside the box → True
Log every verifier decision audit_record appends {**asdict(sample), "suspected_hack": …} to artifacts/verifier.jsonl
Score code by tests instead of exact match CodeVerifier.verify verifier.verify(code, test_cases) → passed / len(tests)
Execute one candidate program CodeVerifier._run subprocess.run([sys.executable, "-c", code], input=…, timeout=5)
Clean markdown fences off generated code strip_fences strip_fences(simple_complete(CODING_TASK))
Turn audited groups into training data build_preferences build_preferences(groups, records) → {prompt, chosen, rejected}

The one sentence that anchors everything: the verifier's quality is the ceiling on the policy's quality — once you train against a checker, the checker is the objective. Every question and the activity reward reasoning about how a checker can be fooled, not about how to get a high score.

2. The Big Picture — the RLVR loop

One prompt fans out into a group of completions; a deterministic program — not a human, not a model — checks each against known ground truth and emits a scalar; audited groups become training data; the policy updates; repeat.

flowchart TD
    P["Problem<br/>question + ground-truth answer"] --> S["Sample N completions<br/>simple_complete · temperature 1.0"]
    S --> G["Group of N Samples"]
    GT[("Ground truth<br/>Problem.answer · code_tests")] --> V
    G --> V{{"VERIFIER — a deterministic program<br/>extract_number → _normalize → compare<br/>CodeVerifier.verify → passed/len(tests)"}}
    V --> R["Scalar reward<br/>+1.0 / -0.1 (binary) · 0.0–1.0 (fractional)"]
    R --> H["looks_like_hack → audit_record<br/>artifacts/verifier.jsonl (append-only)"]
    H --> D["build_preferences<br/>chosen = correct AND unflagged<br/>rejected = incorrect"]
    D --> U["Policy update<br/>DPO pairs · GRPO group rewards"]
    U -.->|next iteration| S

    RM["RLHF instead puts a <b>learned neural reward model</b> here —<br/>trained on human preference comparisons.<br/>Approximate · expensive · non-reproducible · over-optimizable."]
    RM -.->|"RLVR deletes this box"| V
Loading

ASCII fallback:

                      ground truth (Problem.answer / code_tests)
                                        |
   prompt --> sample N --> group --> [ VERIFIER: deterministic program ] --> scalar reward
                 ^                    extract_number | CodeVerifier.verify        |
                 |                                                                v
                 |                        looks_like_hack --> audit_record --> verifier.jsonl
                 |                                                                |
                 +------- policy update <-- build_preferences <--------------------+
                          (DPO pairs / GRPO group rewards)

   RLHF, for contrast:
   prompt --> completions --> [ LEARNED reward model: a neural net fitted to
                                human preference labels ] --> scalar reward --> policy update

Why this shape? The only box that changes between RLVR and RLHF is the one that produces the scalar — and it changes from a model you trained to a program you wrote. Everything downstream (groups, advantages, preference pairs, the policy update) is identical. That single swap is what makes the signal cheap, objective and reproducible — and what makes it a target: a program has blind spots you can read, version and harden, which is why hack detection (§8) and the audit trail sit inside the loop rather than beside it.

3. Setup & the component roles

uv sync          # from the 16_RLVR folder, then select the uv-created Python/Jupyter environment

No GPU (unlike Session 15). One key: OPENAI_API_KEY, entered via getpass in Cell 4 and kept in memory for the session only. The sampling loop makes a few dozen small-model calls (roughly cents). ⚠️ Task 6 executes model-generated Python in a subprocess on your machine — read §10 first.

Role Where it lives What it does
Policy simple_complete / MODEL = "gpt-4.1-nano" the model being sampled; small on purpose at temp 1.0 so some answers are wrong
Verifiable domain Problem, problems 5 math word problems, each carrying a ground-truth answer string
Math verifier extract_number + MathRewardFunction binary reward: exact match after numeric normalization
Code verifier CodeVerifier + code_tests fractional reward: fraction of unit tests passed
Hack detector looks_like_hack flags verified-correct samples showing no visible work
Audit trail AUDIT_LOG → artifacts/verifier.jsonl append-only record of every verifier decision
Training data build_preferences → artifacts/preferences.jsonl {prompt, chosen, rejected} for DPO-style trainers

16_RLVR/.gitignore ignores artifacts/, so verifier.jsonl and preferences.jsonl stay out of the repo. The README's step 4 is the reason: keep the cell outputs (verified-correct rates, flagged counts, a preference-pair example) — those are the durable evidence.

4. What makes a reward verifiable (Breakout Room #1)

A reward is verifiable when a deterministic program — not a human and not a model — decides whether a completion is correct, because ground truth is known and checking it is mechanical. The notebook states the contrast directly (Cell 0):

"Unlike RLHF, there is no learned reward model and no human labeler in the loop. The reward comes from a deterministic program." … and Cell 17: "cheap, objective, reproducible."

Two such programs exist in this notebook:

Verifier Ground truth Verdict
MathRewardFunction.compute Problem.answer (a string) binary: +1.0 / -0.1
CodeVerifier.verify code_tests (input → expected stdout) fractional: passed / len(tests)

RLVR is narrow by construction — it only exists where a checker exists. Question #1 asks you to say what that boundary is and why it is a boundary in principle, not just in practice.

📄 Tülu 3 (coined "RLVR") · DeepSeek-R1

5. \boxed{} extraction and exact match (Task 2)

The prompt instructs the policy to put its final answer in \boxed{} — the GSM8K convention — so extraction is a regex, not a judgment call. Read both branches carefully; they behave differently:

boxed = re.search(r"\\boxed\{([^}]+)\}", text)   # branch 1: the FIRST box wins
if boxed: return boxed.group(1).strip()
numbers = re.findall(r"-?\d+\.?\d*", text)       # branch 2: fallback = the LAST number in the text
return numbers[-1] if numbers else ""            # branch 3: nothing numeric at all

Then _normalize compares numerically when it can (float(value)), and falls back to value.strip() when float() raises. Cell 9 flags the design risk out loud: "a verifier that fails on formatting technicalities punishes correct reasoning, which is the fastest way to teach a policy the wrong lesson." The failure runs in both directions — too lenient rewards non-answers, too strict punishes right answers. Hold that thought for Question #3.

📄 Python re · GSM8K

6. The reward function and its asymmetry (Task 3)

class MathRewardFunction:
    correct_reward = 1.0
    incorrect_penalty = -0.1     # NOT -1.0 — Question #2 asks what -1.0 would teach

Three facts worth having at your fingertips, all asserted in Cell 10:

  • compute(r"The speed is \boxed{80}", "80.0") == 1.0 — normalization makes "80" match "80.0".
  • compute(r"The speed is \boxed{81}", "80") == -0.1 — a wrong answer.
  • compute("I cannot solve this.", "80") == -0.1 — a refusal is not free. extract_number returns "", which fails normalization and earns the same penalty as a wrong answer, not 0.0.

Cell 17 states the design intent: "Asymmetric rewards (+1.0 / −0.1) keep early training from collapsing into refusal." Question #2 asks you to supply the mechanism behind that sentence.

7. Groups, not single samples (Task 4)

groups = [sample_and_verify(p) for p in problems]      # 5 problems x n_samples=4 -> 20 Samples

Each Sample carries problem, response, extracted, reward, verified_correct (= reward > 0). Sampling groups rather than one completion per prompt does two jobs at once:

        one prompt
             |
   +---------+---------+---------+          GRPO  : advantage = reward - group mean  (Session 15)
   v         v         v         v          DPO   : pair a correct winner with an incorrect loser
 sample    sample    sample    sample               ... both need the SAME group structure
  +1.0      -0.1      +1.0      -0.1

Cell 15's NOTE is the operational guardrail: "If your verified-correct rate is 100%, the contrast that drives learning is missing — swap in a smaller model, raise the temperature, or add harder problems until some samples fail." Cell 16 then prints one verifier-rejected completion — reading failures is how you learn what the policy actually gets wrong.

8. Reward hacking, detection, and the audit trail (Task 5, Breakout Room #2)

Goodhart's law — "when a measure becomes a target, it ceases to be a good measure." Cell 19 names three routes the notebook expects: a bare boxed answer with no reasoning, parroting numbers from the prompt, and exploiting extraction quirks instead of solving the problem.

def looks_like_hack(sample: Sample) -> bool:
    if not sample.verified_correct: return False          # only CORRECT samples can be "hacks"
    work = sample.response.replace(f"\\boxed{{{sample.extracted}}}", "")
    numbers_in_work = re.findall(r"-?\d+\.?\d*", work)
    return len(sample.response.split()) < 20 or len(numbers_in_work) < 2

One signature, two thresholds — and a detector is itself a measure. audit_record then appends every decision (flagged or not) to artifacts/verifier.jsonl in append-only mode, so re-running the notebook grows the file rather than replacing it. Question #3 asks for two other hacks plus mechanism-matched hardening.

9. The code verifier — fractional rewards (Task 6)

def verify(self, code: str, test_cases: list[dict]) -> float:
    passed = 0
    for tc in test_cases:
        try:
            output = self._run(code, tc.get("input", ""))
            if output.strip() == str(tc["expected"]).strip(): passed += 1
        except Exception:
            pass                       # crash, timeout, or non-zero exit -> no credit for THIS case
    return passed / len(test_cases) if test_cases else 0.0

Note the shape difference: math is binary (+1.0 / -0.1), code is fractional (0.0–1.0). code_tests has three visible fixtures (1→1, 3→14, 10→385), and strip_fences cleans markdown fences off the candidate before execution. Question #4 asks what partial credit buys and costs.

📄 TRL DPOTrainer · TRL GRPOTrainer

10. Executing untrusted code — the threat model (Task 6 ⚠️)

result = subprocess.run(
    [sys.executable, "-c", code],          # a FULL Python interpreter
    input=input_data, capture_output=True, text=True,
    timeout=self.timeout_seconds,          # = 5
)

Read that call argument by argument: it runs as your user, in your working directory, with your environment (Cell 4 just put OPENAI_API_KEY there) and your network. Cell 22 and Cell 33 say it plainly:

"in production, this verifier runs inside a sandbox (container, gVisor, firecracker VM) — never on the host" … "the policy will eventually generate code that reads the filesystem, opens sockets, or fork-bombs the host (and our timeout catches none of that)" … "the verifier defines the reward, so the verifier's execution environment is a security boundary."

The precise question — which dimensions of that threat does timeout=5 actually constrain, and which does it not touch at all? — is the second half of Question #4.

📄 subprocess.run · Vercel Sandbox · gVisor · Firecracker

11. Preference pairs — the output of the whole pipeline (Task 7)

flagged_responses = {r["response"] for r in records if r["suspected_hack"]}
winners = [s for s in group if s.verified_correct and s.response not in flagged_responses]
losers  = [s for s in group if not s.verified_correct]
# cross product WITHIN a group -> {"prompt": …, "chosen": …, "rejected": …}

Two consequences fall straight out of the code:

  • A group with no incorrect sample (or no unflagged correct one) contributes zero pairs. Cell 28 has an explicit else-branch for exactly that outcome — it is not a bug.
  • Flagged samples are excluded from chosen on purpose: "a hack-suspect completion used as 'chosen' would teach the next policy iteration to hack more."

Same verifier, two consumers: DPO eats the {prompt, chosen, rejected} pairs; GRPO (Session 15) skips the pairing and uses the group rewards directly.

📄 DPO paper

12. Common Issues

Issue Cause Fix
Verified-correct rate is 100% policy too strong / problems too easy for this setup Cell 15's NOTE: smaller model, higher temperature, or harder problems — you need failures
0 preference pairs printed a group needs ≥1 correct and ≥1 incorrect sample Not a bug — Cell 28's else-branch anticipates it. Get contrast first (row above)
artifacts/*.jsonl missing from your repo 16_RLVR/.gitignore ignores artifacts/ Expected. Keep the cell outputs as evidence (README step 4)
Audit trail record count keeps climbing AUDIT_LOG.open("a") is append-only across re-runs Expected. Delete artifacts/verifier.jsonl to reset
A crashing candidate silently scores 0 verify catches bare Exception around _run By design — crash/timeout/non-zero exit = no credit for that case, not an error
Code candidate scores 0.00 but looks right markdown fences, or extra printed text strip_fences; the task says print only the number
Correct answer scored -0.1 \boxed{80 km/h} / 1,250 → float() raises → string compare fails Verifier brittleness, not student error — exactly what Cell 9 warns about
Numbers differ on every run live sampling at temperature 1.0 on a small policy Report the shape of the evidence, never memorized decimals
getpass prompt reappears the key is in-memory for the session only (Cell 4) Re-enter it; never hardcode it into the notebook

13. The four questions — how to get there (no answers, on purpose)

Answer all four in the notebook, in the ❓ markdown cells under ##### Answer:, in your own words.

# Where The skill it's testing Concept section
Q1 after Task 3 what "verifiable" means, and the RLVR ↔ RLHF boundary §4
Q2 after Task 3 reward shaping in the failure-dominated early regime §6, §7
Q3 after Task 5 adversarial reading of your own verifier §5, §8
Q4 after Task 6 sparse vs dense signal, and a real threat model §9, §10

Question #1 — what makes a reward "verifiable"; RLVR vs. RLHF's learned reward model

In your own words: what makes a reward "verifiable," and how does RLVR differ from RLHF's learned reward model? Give one task where a verifiable reward exists and one where it fundamentally cannot (and explain why).

Getting there. The notebook's intro hands you one sentence of definition — push past it. Who or what issues the verdict, and what has to already be known for a verdict to be possible at all? Now list the properties that follow from "a program decides": what does that buy you in cost, in repeatability six months from now, and in your ability to prove to a third party how a sample was scored? (Look at what artifacts/verifier.jsonl exists for.)

For the RLHF half, be precise about what is actually in the loop. Is a human scoring each rollout, or is a model being scored against — one that was fitted to human comparisons? If it's a fitted approximation, ask the uncomfortable question: what happens to its scores when the policy improves and moves off the data it was fitted on, and what does an optimizer do with that?

For "fundamentally cannot," the trap is picking something that is merely hard. Test your candidate against three questions: is there a single ground truth? Would two competent experts agree on the verdict? Is the criterion a computable predicate, or is it defined by preference? If you reach for an LLM judge as the counter-example's rescue — is a second model's opinion deterministic and reproducible, and does calling it a "verifier" survive that test?

One more, worth a sentence: is RLVR immune to reward hacking? Task 5 exists for a reason — check your claim against it before you write it down.

Question #2 — why −0.1 and not −1.0

The reward is asymmetric: +1.0 for a correct answer but only −0.1 for an incorrect one. Suppose we used −1.0 instead. What behavior might a policy learn during early training, when most of its attempts fail? (Hint: think about a model that discovers it can hedge, refuse, or produce no parseable answer at all.)

Getting there. Do the arithmetic before you write prose. Write the expected reward of attempting once as an expression in the success probability p and the failure penalty c. Solve for the p at which attempting becomes worth it — once with c = 0.1, once with c = 1.0. Cell 5 tells you which side of those thresholds an early, deliberately-small policy sits on.

Then switch from economics to gradients. An update pushes probability mass away from whatever earned the penalty. In a regime where most attempts fail, away from what, exactly — and what else shares tokens with those trajectories? Name the concrete cheapest behaviors that stop earning penalties (Cell 12's hint lists three; a fourth is about output length and entropy).

Then follow the damage downstream — this is what separates a partial answer from a full one. If a group contains no verified-correct sample, what does build_preferences return? And under GRPO, what is an advantage worth when every reward in a group is identical?

Finally, a code check rather than a guess: what does extract_number("I cannot solve this.") return, and what reward does Cell 10 assert for it? Does that change the story you were about to tell about refusal being "free"? Say what it does and doesn't change.

Question #3 — two more ways to hack a \boxed{} verifier, and how to harden

Our detector flags one signature: "right answer, no visible work." Name two other ways a policy could hack a \boxed{} exact-match verifier, and for each, describe how you would harden the verifier or the prompt against it. (Session 15's stacked format rewards are one relevant hardening example.)

Getting there. Reread extract_number (§5) line by line as an attacker. It has two independent branches — the first \boxed{} wins, and when there's no box the last number in the text wins. Design one exploit per branch, then ask which one a policy would stumble into by accident rather than by design.

Now reread _normalize. What does float() do with "1,250"? With "80 km/h"? With a unicode minus? Sort the results into two piles: cases where a policy gains reward it didn't earn, and cases where correct reasoning loses reward. Cell 9 warns about the second pile — both are verifier failures.

Now reread looks_like_hack (§8) and count. Exactly what would you have to emit to make both thresholds false? Is that expensive? A detector is a measure too.

Two rules for the hardening half: (1) each fix must defeat that specific mechanism — "make the verifier better", "train longer", "use a bigger model" are not fixes; (2) "penalize wrong answers more heavily" is disqualified by your own Q2 answer, so check it against that before you write it. And the named hint: Session 15's stacked format rewards add a separate reward term for structure. What does paying separately for a parseable answer and a correct one buy you that a stricter correctness check cannot?

Question #4 — fractional vs. binary rewards, and the host-execution threat model

The code verifier returns fractional rewards (fraction of tests passed) while the math verifier is binary. What are the benefits and risks of partial credit as a training signal? And concretely: what could a policy-generated program do to a verifier that runs candidates directly on the host, and which parts of that threat does our timeout not cover?

Getting there — part 1 (the signal). Run the thought experiment on paper. Imagine sampling 8 completions for a hard task under a binary verifier and write down the 8 rewards. What is each sample's advantage relative to the group mean? What does build_preferences return for that group? Now redo it with passed / len(tests) and compare. That contrast is the benefit — name it with the right vocabulary, not just "it's more informative."

For the risks, look at the actual fixtures in Cell 25. What is the cheapest program that scores well on them without solving the stated task? Then ask a second question: if you added three more easy or near-duplicate test cases, whose score would go up — and did the policy get any better? What is the reward now really measuring? (And a third: what happens to reproducibility if a test is flaky?)

Getting there — part 2 (the host). Reread _run (§10) argument by argument. Which user does it run as? Which working directory? Which environment — and what did Cell 4 put into that environment three tasks ago? What else on your machine is readable by that user? Give at least one attack with its mechanism, not a vibe. Then ask the pointed one: what is the one file in this pipeline whose whole purpose is to prove the run was honest, and what happens if generated code can write to it?

Then answer the "timeout" half systematically. Make a list of the dimensions along which a process can do damage — time, CPU, memory, disk, file descriptors, filesystem, network, credentials, child processes, persistence — and mark which single one timeout=5 constrains. For each of the rest, say why it slips through. Two specifics worth checking rather than assuming: how long does reading a key and POSTing it somewhere actually take relative to 5 seconds, and does killing a child also kill a grandchild it started with start_new_session=True? Finally: can any timeout undo a file that was already written?

14. Activity #1 — Build Your Own Verifier (how to approach)

The deliverable: the ### YOUR CODE HERE cell after Task 7, filled in and executed with output kept — a reward function for a new verifiable domain, a sample-and-verify run over at least 3 prompts with n_samples >= 3, a printed verified-correct rate, and a note on hack-suspect behavior and how you'd detect it. The notebook suggests JSON-schema conformance, SQL vs a reference query, or a regex/string transformation; anything genuinely checkable qualifies.

Work the checklist — the discipline is what's being assessed, not cleverness:

  • Pick a genuinely new domain. Not the notebook's math or code verifier reskinned. Sanity test: does your domain have ground truth that exists before the policy answers?
  • Write down your ground truth before you write any code. What object holds it? Can a program compare a completion against it with no judgment call? If your honest answer is "I'd ask a model whether it looks right," stop — §4 says why that isn't a verifiable reward.
  • Design your extraction step — the \boxed{} of your domain. What exactly is the parse, and what does it return when the policy replies with prose, or fences, or nothing usable? (Look at how strip_fences and extract_number each handle their failure case.)
  • Mirror the notebook's reward shape: one completion + ground truth → one float. Keeping MathRewardFunction.compute's signature makes everything downstream reusable.
  • State AND justify binary vs. fractional. The activity asks for the justification explicitly — it is a graded item on its own. Either choice can be right; your reason should connect to what you worked out in Q4 about sparse gradients and about farming the easy checks.
  • Decide what an unparseable output earns. Before you default to 0.0, check what MathRewardFunction gives a refusal, and re-read your own Q2 reasoning about why.
  • Run it for real. ≥3 prompts × n_samples >= 3 — at least 9 actual simple_complete calls against the policy. Hand-written "completions" or a hardcoded rate don't count.
  • Print a verified-correct rate (or a per-group breakdown), not just a list of raw rewards. Be explicit about what counts as "correct" in your domain — is it reward > 0, or must every check pass?
  • Write the hack-suspect observation. The question that gets you there: which part of my reward is the cheapest to earn without actually doing the task? Name a signature specific to your domain and say how you'd detect it in code. "I saw no hacks" only counts if you state the signature you looked for and the method you'd use.
  • Optional depth (not required, but it's where the session's ideas connect): log your decisions into the same AUDIT_LOG, or build preference pairs from your new domain's groups the way Task 7 does — and see whether your flagged samples change the result.

⚠️ A low verified-correct rate is not a failure here — Cell 15's NOTE says a policy that never fails produces no signal at all. Reporting an honest low rate with a clear-eyed observation beats a tidy 100%.

15. References

  1. Tülu 3 — the paper that coined "RLVR". https://arxiv.org/abs/2411.15124
  2. DeepSeek-R1 — RL with verifiable rewards at scale. https://arxiv.org/abs/2501.12948
  3. DPO — Direct Preference Optimization. https://arxiv.org/abs/2305.18290
  4. TRL — DPOTrainer (the {prompt, chosen, rejected} format). https://huggingface.co/docs/trl/dpo_trainer
  5. TRL — GRPOTrainer (the Session 15 consumer). https://huggingface.co/docs/trl/grpo_trainer
  6. GSM8K — the grade-school-math benchmark behind the \boxed{} convention. https://arxiv.org/abs/2110.14168
  7. OpenAI — Chat Completions API. https://platform.openai.com/docs/api-reference/chat
  8. Python — subprocess.run (and what timeout does). https://docs.python.org/3/library/subprocess.html#subprocess.run
  9. Python — re (regex extraction). https://docs.python.org/3/library/re.html
  10. Python — dataclasses (Problem, Sample, asdict). https://docs.python.org/3/library/dataclasses.html
  11. Vercel Sandbox — ephemeral microVMs for untrusted LLM-generated code. https://vercel.com/docs/vercel-sandbox
  12. gVisor — application kernel sandbox. https://gvisor.dev/docs/
  13. Firecracker — microVMs. https://firecracker-microvm.github.io/
  14. SymPy — symbolic equivalence for hardening a math verifier. https://docs.sympy.org/latest/index.html
  15. uv — the project/dependency manager used by uv sync. https://docs.astral.sh/uv/
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment