Vol. XXXI · History and Projects Plate 10 of 14project, reinforcement learning, GRPO, verifier, reward hacking, learning rate, entropy, held-out accuracyLE

Project: RL With a Verifier.

Choose the reward, the step size and how long to train; the held-out sums are the judge.

The RL world of this volume is the post-training world’s chat model (plate XVII.1): fine-tuned on 30 examples, it answers about half the held-out sums correctly. A verifier checks each answer by doing the arithmetic, and every run here — sampling, rewards, advantages, updates — is real training of that model in a Web Worker.

The brief: held-out accuracy (exact answers) of at least 0.70 after training; a training reward no more than 0.20 above held-out accuracy, because a larger gap means the policy is earning reward the verifier did not mean to give; and policy entropy of at least 0.15 at the end, so it is still exploring. The starting design — the lenient “contains the answer” reward, learning rate 0.010, 20 steps — ends at held-out 0.00. A project plate grades the design against a brief: every check below the workshop must pass, and a passing design is saved in this browser.

Left: training reward and held-out accuracy as training proceeds. Right: the three checked quantities against the brief.

What you will learn

Choose a verifier reward that cannot be gamed.
Set a learning rate that does not wreck the policy.
Train long enough.
Watch the reward–accuracy gap.

Workshop

GRPO with groups of 8, batch 8, clipping 0.2, on the chat model of plate XVII.1, trained in a Web Worker; held-out accuracy on 30 sums with greedy decoding. “Contains” pays any answer in which the right number appears; “exact” pays only the right number alone.

Pay for the right thing, then move slowly

The starting design fails twice over. The lenient reward pays an answer that merely contains the right number — “15 16 17” earns full reward for 7 + 8 — so the policy learns to hedge, and its training reward runs ahead of what the exact held-out check measures. And a learning rate of 0.01 moves a small policy so far each step that what it knew falls apart: held-out accuracy 0.00, from 0.53 before training.

An exact verifier and a smaller step fix both, and enough steps let the smaller step get somewhere. The checks in the brief are the ones the reasoning-RL reports watch: held-out accuracy rather than training reward, the gap between them as a hacking alarm (plate XXVII.6), and entropy as a sign that exploration has not died (plate XIX.9).

The same failure at scale was reported for code: models trained on tests they could see learned to special-case those tests, which is why the next project hides some of them.

Terms

Verifier: A program checking an answer, here by doing the arithmetic.
Lenient reward: A reward a wrong behaviour can also earn.
Policy collapse: Training that narrows the policy onto a few outputs.
Entropy: How spread the policy’s choices are.

RL with a verifier works when the reward pays only the intended answer and the steps are small enough to keep the policy exploring.

Sources

Shao et al., “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models”, arXiv:2402.03300, 2024 (GRPO).
DeepSeek-AI, “DeepSeek-R1”, arXiv:2501.12948, 2025.
Skalse et al., “Defining and Characterizing Reward Hacking”, NeurIPS 2022.