Drills: RL for Reasoning.
Five questions at a time from the whole volume, until a reasoning model’s training recipe reads like a recipe.
This plate draws rounds of five from the questions and predictions of the eleven plates before it, with the reasoning and source plate after each answer.
The volume’s thread is learning from one’s own attempts: a verifier turns samples into a signal, the policy gradient turns the signal into updates, and the design questions are about where the signal is rich — varied samples, mixed groups, checked steps, honest verifiers.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Frame a language model as a policy.
- Compute policy gradients and advantages.
- Compare PPO, GRPO and rejection sampling.
- Read pass@k and test-time scaling.
What to carry forward
A language model is a policy whose expected reward is its chance of a right answer (plate XIX.1); REINFORCE with a baseline climbs it (plate XIX.2), and verifiers supply rewards that are exact about what they check (plate XIX.3). PPO reuses samples under a clipped ratio (plate XIX.4); GRPO uses group baselines and loses uniform groups (plate XIX.5); rejection sampling learns from kept successes (plate XIX.6).
Chains of thought buy serial computation (plate XIX.7), process rewards solve credit assignment if steps can be judged (plate XIX.8), strong RL can sharpen without widening (plate XIX.9), test-time compute trades inference for accuracy (plate XIX.10), and distillation copies a teacher’s filtered working (plate XIX.11). Vol. XX asks what all this training should aim at.
Terms
- REINFORCE
- E[(R − b) ∇ log π].
- GRPO
- Group-relative advantages.
- pass@k
- At least one of k right.
- Chain of thought
- Steps written before the answer.