Session 15 Cheat Sheet — Reasoning Model Fine

session15-cheatsheet-reasoning-model-finetuning.md

Session 15 Cheat Sheet — Reasoning Model Fine-Tuning with GRPO

A frame to help you reason through the assignment — the mental model, the loop diagram, and the map of which cell does what. It deliberately does not contain the answers or write-ups for you. Instead it gives you the questions to ask yourself and the cells to go inspect. The work — and the learning — is in running the training, watching the reward move, and writing your own conclusions.

Notebook: 01_Reasoning_Fine_Tuning_Unsloth_GRPO.ipynb Model: meta-llama/Llama-3.2-3B-Instruct (gated — accept the license and hf auth login, or use the ungated unsloth/ mirror). Dataset: openai/gsm8k. One idea to hold onto while you read: we never show the model examples of good reasoning — we only reward it. Keep asking yourself what, exactly, is being rewarded, and what that pressures the model to become.

---

1. Quick Reference

You want to…	Reach for	Where to look
Load base model (16-bit)	FastLanguageModel.from_pretrained	Task 2 — note the load_in_4bit value and ask why
Attach LoRA adapters	get_peft_model	Task 3 — then read print(model) (you need it for Q2)
GSM8K → prompts + gold	get_gsm8k_questions()	Task 4 — what does the system prompt force? what does #### mark?
Score a completion	5 reward functions	Task 5 — which one is worth the most, and why do the others exist?
Configure the RL run	GRPOConfig	Task 6 — num_generations, warmup_ratio, learning_rate, scheduler
Train	GRPOTrainer(...).train()	Task 7 — watch the reward column
Generate base vs tuned	fast_generate(lora_request=...)	Task 8/9 — None vs the saved adapter

One sentence to keep in mind: GRPO scores each completion relative to the group's average reward. That single design choice is what lets it skip a piece of machinery PPO needs — figure out which piece, and why the group average can stand in for it. That's the heart of the video explanation.

---

2. The Big Picture — the GRPO loop

flowchart TD
    P[GSM8K prompt] --> G[sample a GROUP of completions<br/>num_generations=?]
    G --> R[score each with the reward stack]
    R --> A[group-relative advantage<br/>reward vs group average]
    A --> U[policy update<br/>+ KL penalty]
    U --> P
    U -.only the adapter moves.-> W[which weights actually change?]

Loading

ASCII fallback:

prompt ─► sample a group of completions ─► reward each ─► advantage = reward vs GROUP AVERAGE
   ▲                                                                    │
   └─────────────── policy update (only the LoRA matrices?) ◄───────────┘

Orientation, not answer: the notebook's intro (Cell 0) states the loop in five arrows and makes one claim about why no value network / critic is needed. Find that claim and make sure you can say it in your own words — it's exactly what the video asks for. Then, as you read Tasks 3–9, keep a running answer to: which weights actually change during training, and which stay frozen?

---

3. Setup & the hardware gate

uv sync          # from 15_Reasoning_Model_Fine_Tuning; Linux/WSL2 + Ampere+ GPU only
hf auth login    # Llama-3.2 is gated — or swap to the unsloth/ mirror in the notebook

This session trains a model on your own GPU — Ampere+ (compute ≥8.0), 16GB+ VRAM, Linux/WSL2. If you hit out-of-memory, the README lists the knobs (gpu_memory_utilization, num_generations, gradient_accumulation_steps). Before you change num_generations, read Task 6's note about which two numbers have to stay divisible — changing one without the other is the classic crash. No suitable GPU? Use the ColabVersion__…ipynb on a Colab-Pro L4/A100.

---

4. Core concepts — investigate, don't skip

These are what Questions #1–4 ask you to explain in your own words. Below is where to look and what to compare — not the conclusion. Form your own read first, then check it against the notebook's callouts.

16-bit vs 4-bit (→ Question #1)

Find where Task 2 sets load_in_4bit and read Cell 0's note on why this session trains 16-bit. Then recall the QLoRA paper's double quantization: what exactly gets quantized the second time — the weights again, or something else the 4-bit packing created? Ask: given that GRPO spends most of its time generating 8 completions per step, what would 4-bit cost you on that path? And flip it — when would 4-bit (QLoRA) be the right call?

What LoRA changes vs. leaves alone (→ Question #3)

Read Cell 8's description and the get_peft_model call in Task 3. For a weight matrix W, what two small matrices get introduced, and what is the effective weight during training? Which parameters receive gradients — all of W, or just the new pieces? Write your own one-sentence definition of LoRA before you read the notebook's, then compare.

Reading print(model) (→ Question #2)

Run Task 3 and study the printout. The diagram in Q2 gives you Layer Norm as a worked example (LlamaRMSNorm). For each remaining label — Feed Forward, Masked Multi Self-Attention, Text & Position Embed, Text Prediction — find the matching module name in the printout. One of them is a trap: does Llama store positions in a learned embedding table, or does it get position information some other way? Check where lora_A/lora_B appear and connect that back to target_modules.

What the reward stack rewards (→ Activity #1, and the video)

Read Task 5's table and the five functions. Which single reward is worth the most? The others give only 0.5 or less — so why include them at all? Ask yourself: at step 0, before the model ever gets an answer right, is there anything it can do to earn a non-zero reward? What would training look like if the only reward were correctness?

The training knobs (→ Question #4)

Open the GRPOConfig in Task 6. For warmup_ratio, learning_rate, and lr_scheduler_type: which one sets how big each step is, which sets how the step size ramps up at the start, and which sets how it decays afterward? The TRL TrainingArguments docs (linked in Q4) are fair game. Then ask why an RL fine-tune in particular wants a gentle warmup.

Before vs. after (→ Activity #2)

Read Task 8's note: vLLM is still holding the frozen base weights, so the same engine can generate two ways. What argument flips generation between the base model and your trained adapter? Plan to run the same prompt both ways so your comparison is controlled.

---

5. Questions — what to ask yourself (write your own answers in the notebook)

Q1 — Double quantization & why 16-bit here

In the QLoRA scheme, what does 4-bit packing create that itself costs memory — and is that what double quantization compresses? Roughly how much does it save per parameter? Then: given GRPO samples 8 completions every step, why does this notebook stay 16-bit — and when would you still pick QLoRA?

Q2 — Label the decoder

Match each diagram label to a module in your print(model) output. Which block holds gate/up/down? Which holds q/k/v/o? Is "Text & Position Embed" really two learned tables here, or does Llama get positions another way? Which module turns the final hidden state into vocabulary logits?

Q3 — What is LoRA doing?

For a frozen weight W, what does LoRA add, and what is the effective weight while training? Which parameters actually update? Why does this make fine-tuning a 3B model feasible on one GPU? (Careful not to describe QLoRA's quantization here — that's a different idea.)

Q4 — warmup_ratio / learning_rate / lr_scheduler_type

Which of the three is the step size, which is the ramp-up, and which is the decay shape? Tie each to training stability: what goes wrong with no warmup, or with too large a learning rate, on a noisy RL signal?

---

6. Activities — the deliverable + how to get there

Activity 1 — Run GRPO training and keep the reward logs

Deliverable: an executed GRPOTrainer.train() (Task 7) with the reward logs kept (README: "especially the training reward logs"), plus your own note on what you observed. Method: let it run and watch the reward column. Cell 27 warns about an "Aha" moment — reward hovers near ~0, then climbs. Keep evidence of the trajectory, not just a final number. The correctness_reward_func prints a sample completion each batch — watching those go from rambling to structured is exactly the story to capture. Modest movement in a short run is expected; don't chase a big number.

Activity 2 — Before/after comparison

Deliverable: the same prompt run through the base model and your fine-tuned adapter (Task 8/9), plus a written observation on what changed. Method: generate once with the base (frozen weights) and once with your trained adapter, using the same prompt so it's a fair comparison. Describe the behavior difference — format, structure, reasoning — not just "the second one is better." Ask yourself: did the format change, the correctness, or both?

---

The learning is in running the training, watching the reward actually move, comparing base vs. tuned on the same prompt, and writing your own conclusions. The notebook's callouts (Cell 0's loop, Cell 27's "Aha" note, the Breakout Room summaries) are there to check your reasoning against — read them after you've formed your own answer, not instead of forming one.

session15-notebooklm-briefing.md

Reasoning Model Fine-Tuning: Group Relative Policy Optimization (GRPO) with Unsloth

Executive Summary

The following document outlines a technical framework for fine-tuning large language models (LLMs) into reasoning models using Group Relative Policy Optimization (GRPO). By applying Unsloth’s implementation of GRPO—the reinforcement learning (RL) algorithm utilized by DeepSeek-R1—the process transforms a base model (meta-llama/Llama-3.2-3B-Instruct) into one capable of structured, step-by-step reasoning for grade-school math problems (GSM8K).

Critical Takeaways:

• Algorithm Efficiency: Unlike standard Reinforcement Learning from Human Feedback (RLHF), GRPO eliminates the need for a separate value network (critic), making it practical for small-scale hardware.
• Training Methodology: The approach moves away from Supervised Fine-Tuning (SFT). The model is never shown how to reason; instead, it is rewarded for correct final answers and specific formatting, discovering reasoning paths through trial and error.
• Hardware Optimization: Using 16-bit LoRA (Low-Rank Adaptation) rather than 4-bit QLoRA is prioritized to maximize generation throughput, which is the primary bottleneck in GRPO training.
• The "Aha!" Moment: Training typically exhibits a "delayed liftoff" where rewards remain flat until approximately step 100–150, at which point the model suddenly begins to converge on successful reasoning strategies.

I. Core Training Framework: GRPO Mechanics

Group Relative Policy Optimization (GRPO) shifts the fine-tuning paradigm from orchestrating models to fundamentally changing model weights through a verifiable reward game.

The GRPO Loop

1. Group Sampling: For every prompt, the policy generates a group of completions (e.g., eight) simultaneously.
2. Reward Scoring: Each completion is evaluated by a stack of reward functions.
3. Group-Based Advantage: Rewards are compared against the group average. Completions performing above the mean receive a positive advantage; those below receive a negative advantage.
4. Policy Update: The model is nudged toward outputs with positive advantages. A KL (Kullback–Leibler) penalty is applied to prevent the model from drifting too far from the original base policy.
5. Iteration: The updated policy samples new groups, repeating the cycle until reward convergence.

Differentiation from SFT and PPO

Unlike Supervised Fine-Tuning (SFT), GRPO does not require reasoning traces or preference pairs. Unlike Proximal Policy Optimization (PPO), it does not require a secondary "critic" model to estimate value, significantly reducing VRAM requirements and complexity.

II. Model Architecture and Parameter-Efficiency

The training utilizes LoRA (Low-Rank Adaptation) to maintain efficiency. This method freezes the base weights of the 3B-parameter model and trains only a small fraction of the weights via low-rank update matrices (A and B).

Technical Specifications

• Precision: 16-bit LoRA is preferred over 4-bit QLoRA. While QLoRA saves memory, GRPO's wall-clock time is dominated by generating completions. vLLM’s fast-generation path is most stable and performant on 16-bit weights.
• Rank (r) and Alpha: A rank of 64 is used for the decomposition, with a scaling factor (alpha) also set to 64.
• Target Modules: Adapters are attached to all linear layers, including the four attention projections and three MLP projections.
• Context Headroom: By setting UNSLOTH_VLLM_STANDBY, training and vLLM share GPU memory, increasing context headroom by approximately 30%.

III. Dataset Preparation and Reward Engineering

The training uses the GSM8K dataset, which consists of math questions and final answers. Because the dataset provides "gold" answers after a #### delimiter, it serves as a verifiable source for automated scoring.

Structured Reasoning Format

To make completions checkable, a system prompt enforces an XML-style structure:

• Reasoning: Contained within ... tags.
• Answer: Contained within ... tags.

Stacked Reward Functions

A suite of functions provides a "gradient for the model to climb," offering partial credit for format before the model masters correctness.

Reward Function Objective Max Reward Correctness Extracted answer matches the gold answer. 2.0 Integer Format Extracted answer is a plain integer. 0.5 Strict Format Matches exact / layout. 0.5 Soft Format Tags appear in the correct order (lenient). 0.5 XML Count Partial credit for correctly placed tags; penalty for trailing junk. ~0.5

IV. Training Configuration and GRPOTrainer

Configuration requires precise batch geometry to ensure sampled groups tile evenly into the effective batch.

Key Parameters

• num_generations (Group Size): 8. This is the number of completions compared within each step.
• Gradient Accumulation: Set to 8 to ensure compatibility with group size.
• Optimizer: adamw_8bit is used to shave several gigabytes of VRAM.
• Learning Rate Schedule: Includes a warmup_ratio to stabilize early training and an lr_scheduler_type (such as cosine) to manage the learning rate over time.
• Max Sequence Length: 2048 tokens, providing headroom for long reasoning traces.

Training Progress

During training, the "reward" metric is monitored rather than "loss." Correctness rewards typically remain near zero for the first 100–150 steps. As format rewards stabilize, the model eventually reaches an "Aha!" moment where the correctness reward begins a sharp ascent.

V. Deployment and Comparison

The final output of this process is a LoRA adapter. One of the primary advantages of this architecture is the ability to swap personalities within the same engine.

• Base Model Inference: The frozen base weights remain accessible by passing lora_request = None to the vLLM engine.
• Trained Model Inference: The reasoning capabilities are activated by loading the saved LoRA adapter.
• Performance Delta: While the base model might provide generic mathematical explanations or code snippets, the fine-tuned model consistently utilizes the and format to solve problems.

Summary of Learning Outcomes

The synthesis of Unsloth, vLLM, and GRPOTrainer allows for the creation of reasoning models that do not rely on expensive human-annotated reasoning traces. Instead, they leverage verifiable datasets and structured rewards to "discover" logic through reinforcement.

session15-notebooklm-glossary.md

Foundations of Reasoning Fine-Tuning: A Technical Glossary

1. The Core Paradigm: Parameter-Efficient Fine-Tuning (PEFT)

Parameter-Efficient Fine-Tuning (PEFT) is a methodology used to adapt large-scale pre-trained models by updating only a small subset of parameters. For the independent developer, this is the "great equalizer." Historically, fine-tuning reasoning models required massive industrial compute clusters. PEFT turns a multi-thousand-dollar compute requirement into a project that runs effectively on a single consumer-grade or mid-range enterprise GPU, such as an NVIDIA L4 or A100.

The primary benefits of PEFT are:

• Memory Efficiency: By limiting the number of trainable parameters, the VRAM required to store optimizer states and gradients is drastically reduced.
• Frozen Base Weights: The original model parameters remain untouched, preserving the "foundational knowledge" and preventing catastrophic forgetting.
• Portability: The output is a lightweight "adapter" file, often only a few hundred megabytes, which can be shared or swapped without needing to move the multi-gigabyte base model.

While PEFT provides the high-level framework for efficiency, we utilize specific mathematical algorithms to handle the actual weight adaptation.

2. Low-Rank Adaptation (LoRA) and Weight Mechanics

The LoRA algorithm is the industry standard for implementing PEFT. It works by freezing the pre-trained weight matrix (W) and injecting two smaller, trainable matrices—Matrix A and Matrix B. Instead of updating every value in the massive W matrix, the model learns a "low-rank" update (W + AB).

Hyperparameter Definition Role in Training Rank (r) The dimensionality of the matrices A and B. Determines the "capacity" of the adapter. Higher rank allows for more complex learning but increases VRAM usage. Alpha A scaling factor for the weight updates. Controls the "influence" of the adapter over the base model. Target Modules The specific layers where adapters are injected. In the Llama-3.2-3B-Instruct architecture, we target all linear projections: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj.

Stability Practice: In practice, setting Alpha equal to Rank (e.g., 64/64) is the preferred standard. This provides a consistent scaling for updates, ensuring that training remains stable across different ranks without requiring extensive re-tuning of the learning rate.

These adapters are injected into the architecture, but their performance and memory footprint are further dictated by the precision level of the weights.

3. Precision and Quantization: 16-bit vs. 4-bit (QLoRA)

Choosing a precision level is a trade-off between hardware accessibility and training throughput.

1. NF4 (NormalFloat 4): A 4-bit data type optimized for weights with a normal distribution, allowing models to fit on significantly smaller GPUs.
2. Double Quantization: A technique that quantizes the quantization constants themselves, saving an additional ~0.37 bits per parameter.

The GRPO Trade-off: While 4-bit (QLoRA) is essential for low-VRAM environments, it can significantly hinder generation speed. Because the GRPO training loop relies on generating a high volume of completions (group sampling), 16-bit precision is the superior choice. It allows the vLLM engine to utilize its optimized "fast-generation" path, which provides the high throughput and stability required for reinforcement learning.

When to Reach for...

• 16-bit LoRA: The default choice for GRPO if you have 16 GB of VRAM or more. It maximizes training speed and leverages vLLM stability.
• 4-bit QLoRA: Use only when memory is the absolute binding constraint, such as fitting a 3B model into an 8 GB VRAM budget.

With the architecture and precision defined, we move to the reinforcement learning loop that teaches the model how to reason.

4. Group Relative Policy Optimization (GRPO)

Group Relative Policy Optimization (GRPO) is the reinforcement learning algorithm behind DeepSeek-R1. It is designed to foster emergent reasoning by rewarding specific outcomes rather than showing the model "correct" steps.

Algorithmic Innovation: Unlike traditional Reinforcement Learning from Human Feedback (RLHF) which requires a separate "value network" (critic model) to estimate rewards, GRPO uses group averages. This eliminates the need for an additional model, saving significant VRAM and making reasoning training viable on smaller hardware.

The Core Loop:

1. Group Sampling: For a single prompt, the model generates a group of multiple completions (e.g., 8 different attempts).
2. Reward Scoring: Each completion is analyzed by a stack of automated reward functions.
3. Group-based Advantage: Each response’s reward is compared to the group’s average. Above-average responses get a positive "advantage."
4. Policy Update: The model's weights are nudged to reinforce the behaviors that led to the high-advantage responses.
5. KL Penalty: A "Kullback–Leibler" constraint acts as a mathematical leash, preventing the model from deviating so far from its original state that it becomes incoherent.

For the model to "climb" this reward gradient, it requires a dataset that can be judged objectively.

5. Verifiable Reasoning and the GSM8K Framework

To train a reasoning model without human labels, we use verifiable answers. The GSM8K (Grade School Math 8K) dataset is the standard here, as math problems have definitive, checkable integer or string results. We do not show the model how to reason; we only reward it when the final answer is correct and the formatting is followed.

Reward Function Objective Max Reward Correctness Matches the extracted answer to the "gold" answer. 2.0 Integer Verifies the extracted answer is a plain integer. 0.5 Strict Format Rewards exact adherence to the required XML layout. 0.5 Soft Format Rewards the correct order of XML tags. 0.5 XML Count Rewards the correct placement and count of tags. ~0.5

To facilitate this, we enforce a specific XML tag structure for the model's output:

[The model's internal step-by-step logic goes here] [The final verifiable answer]

In the early stages of training, the model often fails to produce these tags correctly or hallucinates the format. This results in a period where the model struggles to earn any rewards, setting the stage for the breakthrough in the learning process.

6. The Learning Narrative: The "Aha!" Moment

The experience of training with GRPO differs from standard supervised fine-tuning. Instead of watching "Loss" go down, developers watch "Reward" go up.

The "Aha!" Moment: During the first ~100 steps, the reward curve usually stays flat and near zero. During this phase, the model is slowly learning to master the Format Rewards (0.5). Once the model consistently masters the XML structure, it suddenly has the "Aha!" moment—it discovers that organized reasoning leads to the massive Correctness Reward (2.0). This creates a sharp, non-linear climb in the reward logs as logic and format finally align.

Key Takeaway: The result of this process is not a new base model, but a portable LoRA adapter. This adapter acts as a reasoning "plugin," teaching a standard model to engage in structured chain-of-thought processing. It proves that with the right reinforcement loop, high-level reasoning can be squeezed into lightweight, efficient architectures.