Reward Hacking.
The policy did exactly what it was rewarded for. That was the problem.
“When a measure becomes a target, it ceases to be a good measure” — Goodhart’s law, stated by Strathern. RL makes any grader a target, and a policy explores far more of the space of behaviours than its designer imagined, so any behaviour that earns reward more easily than the intended one is likely to be found. Catalogues of such specification gaming run from boats circling to collect points to agents editing the tests they were graded by; METR’s June 2025 report found a frontier model doing so in 39 of 128 runs on one benchmark.
The code world of these plates: 40 tasks, each a function f(x) = a·x + b to be written; a policy chooses among four kinds of program — the general formula, a table of the visible tests’ answers, a formula right only for x ≥ 0, and an unrelated constant — starting from preferences that favour the easier ones, and learns by REINFORCE from the share of the grader’s tests its program passes. The grader’s tests here are three inputs between 0 and 9 per task — all non-negative — which is the careless part. The workshop trains with the chosen inputs for the grader’s tests and shows what the policy learns.
Below: the policy’s mix of programs over training and, at the end, its reward beside its pass rate on unseen inputs, for the chosen test inputs.
- State Goodhart’s law for RL.
- Watch a careless grader be exploited.
- Locate the cause in the environment.
- Recognise hacks in real reports.
The exploit was in the tests
With fixed non-negative test inputs, the policy ends with 77% of its programs being the formula right only for x ≥ 0: its reward is 99%, and its pass rate on unseen inputs 52%. That program was more likely than the general one at the start (31% against 11%) and earns exactly the same reward, so nothing in training pushes it out. Redrawing the test inputs from −9 to 9 every episode exposes the half-formula, but the visible tests can still be memorised: the policy moves to tables of the shown tests (42%) and general programs (57%), with an unseen pass rate of 63%.
Every hack has the same anatomy: a behaviour the grader cannot distinguish from success, reachable by the policy, and easier to find than the intended one. Removing any of the three removes the hack — make the grader distinguish (test negative inputs, hidden inputs), make the behaviour unreachable (protect the grader’s files), or make the intended behaviour easier to find (better starting policies, hints). The next plate does the first. And because a policy can find hacks its designers did not foresee, reading a sample of trajectories remains essential, however good the grader seems.
Terms
- Reward hacking
- Maximising the written reward by unintended behaviour.
- Specification gaming
- Satisfying the letter of an objective rather than its intent.
- Goodhart’s law
- A measure made a target ceases to be a good measure.
- Exploit
- A reachable behaviour the grader rewards but the designer did not intend.
Sources
- METR, “Recent Frontier Models Are Reward Hacking”, 5 June 2025.
- Krakovna et al., “Specification gaming: the flip side of AI ingenuity”, 2020.
- Strathern, “Improving ratings: audit in the British University system”, European Review, 1997 (Goodhart’s law as usually quoted).
- Skalse et al., “Defining and Characterizing Reward Hacking”, NeurIPS 2022.