Vol. XXVII · RL Environments Plate 2 of 11graders, verifiers, false positives, false negatives, exact match, rubrics, model judgesLE

Graders and Verifiers.

Every wrong answer a grader accepts is a mistake that training will reinforce.

A grader can be an exact comparison with a reference answer, a program that runs tests (plate XXVII.3), a rubric applied by people, or a model asked to judge (plate XXI.5). It can err in two ways: a false positive rewards a wrong episode, a false negative withholds reward from a right one. For training the first is worse: false positives are rewards for mistakes, and RL concentrates the policy on whatever earns them (plate XXVI.9). Exact verifiers have few false positives but exist only for tasks with checkable answers; that is why RL has worked best on maths and code.

The tool world of this part: the character model of Vol. XVII, pre-trained only on single-digit worksheets, is fine-tuned in the page to call a calculator — it writes “B23+45|”, the environment appends the result “68|”, and the model answers “68|”. The workshop samples four episodes per held-out sum at temperature 1 from a partly trained agent (fine-tuned 100 steps on 30 demonstrations), and grades each with four graders: exact final answer; final answer containing the sum’s digits; the sum appearing anywhere after the prompt; and the final answer repeating the last calculator result.

Below: each grader’s false-positive rate (wrong episodes rewarded) and false-negative rate (right ones missed), with an accepted wrong episode for the chosen grader.

What you will learn

Define false positives and negatives of a grader.
Compare four graders on the same episodes.
Explain why false positives matter most.
Relate verifiability to where RL works.

Workshop

120 episodes (4 per held-out sum, temperature 1) from the agent fine-tuned for 100 steps on 30 demonstrations. “Right” means the final answer equals the sum. Contains: the sum’s digits appear in the final answer. Transcript: the sum appears anywhere in the reply, tool results included. Consistent: the final answer equals the last calculator result.

False positives are training signal

Of 120 sampled episodes, 16% are right. The exact grader pays exactly that. The consistency grader pays 77%: it accepts 73% of the wrong episodes, every one in which the agent made a wrong call and faithfully copied the wrong result. The transcript grader accepts 4% of wrong episodes — those where the right sum appeared in a tool result but the agent answered something else — and the contains grader 0%, because this agent’s answers are short. Plate XXVI.9 showed what RL does with the consistency grader’s gap.

Real graders face the same trade-off with more at stake. Exact match on a final answer needs canonical answers; unit tests check behaviour but only what they test; model judges handle open tasks but can be persuaded by confident, long or well-formatted answers. Good practice is to measure a grader before training against it: hand-label a sample of episodes, estimate both error rates, and look specifically for wrong episodes it accepts — those are the behaviours training will grow.

DeepSeek-R1’s early reasoning RL deliberately used rule-based rewards — answer checking and format checks — rather than a neural reward model, citing reward hacking of learned rewards at scale.

Terms

Verifier: A grader that checks correctness by rule or execution.
False positive: A wrong episode the grader rewards.
False negative: A right episode the grader fails to reward.
Rubric: Written criteria a person or model applies to grade.

A grader’s false positives are rewards for mistakes; measure a grader’s errors before training against it.

Sources

Lightman et al., “Let’s Verify Step by Step”, ICLR 2024.
DeepSeek-AI, “DeepSeek-R1”, arXiv:2501.12948, 2025 (rule-based accuracy and format rewards).
Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, NeurIPS 2023.