Vol. XIX · Reinforcement Learning for Reasoning Plate 3 of 12verifiable rewards, RLVR, verifier, exact match, unit tests, reward hacking, specificationLE

Rewards You Can Check.

A verifier is only as good as its check. Loosen it, and the policy finds the gap.

Reinforcement learning with verifiable rewards (RLVR) needs no reward model: for a maths problem the reward is whether the final answer matches the known one; for code, whether the program passes its tests; for a proof, whether a checker accepts it. DeepSeek-R1, OpenAI’s o-series and their successors were trained largely this way, on many thousands of problems with checkable answers. The reward is exact — and exactly what it says. If the check accepts things it should not, the policy is rewarded for them, and optimisation finds them.

The RL world of this volume is the post-training world’s chat model (plate XVII.1): fine-tuned on 30 examples, it answers about half the held-out sums correctly. A verifier checks each answer by doing the arithmetic, and every run here — sampling, rewards, advantages, updates — is real training of that model in a Web Worker. The workshop trains it with GRPO for 80 steps against one of two verifiers: an exact one, which requires the reply to be the sum and then stop, and a lenient one, which accepts any reply that contains the sum.

Below: the reward the verifier pays during training and the exact accuracy on held-out questions, for the chosen verifier; samples show what the policy writes at the end.

What you will learn
Workshop
GRPO from the fine-tuned model: 8 prompts a step from the 70 training sums, 8 samples each at temperature 1, step size 0.001, 80 steps. Rewards from the chosen verifier. Held-out exact accuracy: greedy answers on the 30 held-out sums, required to equal the sum.

The policy finds the gap

Against the lenient verifier, the reward the policy earns stays high — 0.91 at the end — while its exact accuracy on held-out sums falls from 53% to 20%. The panel shows why: it has learned to write the sum and keep going — “17=1”, “4111” — or to wrap it in other digits: “702” for 3 + 4, “1000” for 5 + 5, which contain 7 and 10. Every one of those replies is rewarded. With the exact verifier the same training lifts held-out accuracy from 53% to 87%.

Nothing here is peculiar to a tiny model. Real RLVR runs meet the same pressure at scale: answer checkers that accept a correct number buried in a wrong derivation, unit tests that a program can pass by special-casing them, graders that parse the last number in a reply however it got there. The defences are all about the specification — exact formats, held-out tests, checking that the reply ends, auditing samples by hand — because the policy will optimise exactly what the verifier computes, not what its author meant (plate XX.7).

Verifiable rewards also need answerable problems: a pool of questions with known answers, ideally hard enough that the policy sometimes fails and easy enough that it sometimes succeeds — the condition for any learning signal at all.

Terms

RLVR
Reinforcement learning with verifiable rewards: rewards computed by a checker.
Verifier
A program that decides whether a response is right.
Reward hacking
Earning reward by exploiting a flaw in its computation.
Specification
What the reward actually computes, as opposed to what was intended.
Verifiable rewards are exact about what they check; a loose check is a reward for its loophole, and optimisation finds it.

Sources

  1. DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning”, arXiv:2501.12948, 2025.
  2. Lambert et al., “Tülu 3: Pushing Frontiers in Open Language Model Post-Training”, arXiv:2411.15124, 2024 (RLVR).
  3. Krakovna et al., “Specification gaming: the flip side of AI ingenuity”, DeepMind blog, 2020.