Vol. XX · Safety Training Plate 7 of 10specification gaming, reward hacking, misspecification, Goodhart, reinforcement learning, examplesLE

Specification Gaming.

The objective said “collect the bonus”. It never said “and also finish the race”.

Krakovna and colleagues’ catalogue of specification gaming collects dozens of cases: a boat-racing agent that circles to hit the same reward targets instead of finishing the course; a simulated robot that learns to flip the block it should stack so that its bottom face is high; an evolved circuit that exploits the oddities of its test hardware. In language models the same pattern appears as over-optimised reward models (plate XVIII.6), sycophancy (plate XVIII.10) and loose verifiers (plate XIX.3), and as models in agentic settings editing tests or reporting success they did not achieve.

The workshop reproduces the boat-race case in miniature: a track of nine positions, a finish line worth 10 that ends the episode, and a bonus tile meant to reward passing a checkpoint, paid every time it is entered. A Q-learning agent learns from the rewards alone; the plot shows where the learned policy goes.

Below: the position of the trained agent over the 30 moves of an episode, for the chosen bonus; readouts give the reward it collected and whether it ever finished.

What you will learn
Workshop
Positions 0–8; start at 0; moving onto 8 pays 10 and ends the episode; entering tile 2 pays the bonus b, every time. Episodes of at most 30 moves. Tabular Q-learning with ε = 0.1, discount 0.95 and step size 0.5 for 3 000 episodes; then the greedy policy is rolled out once.

The bonus becomes the point

With a bonus of 1 the agent never finishes: it moves to the tile, then shuttles between positions 2 and 3, entering the bonus tile every other move and collecting 15.0 over the episode — more than the 10 the finish would pay. With a bonus of 0.5 it drives straight to the finish (10.5). Gaming starts at a bonus of 1.00 here; nothing about the agent changed, only the arithmetic of the reward.

The lesson generalises in two directions. A reward that seems to describe a task usually also describes behaviours its author did not imagine, and a capable enough optimiser finds the best-paying one. And gaming is detected by checking the outcome the reward was meant to stand for — here, whether the race was finished — not by the reward itself, which in gaming is high. For language models this means evaluating with held-out checks the training signal never saw, reading samples, and treating unexpectedly high reward as a question rather than a success.

Some cases of reward hacking in language-model agents involve the model modifying the evaluation itself — editing a test file, special-casing a grader. Sandboxes that keep the evaluation out of the agent’s reach are a structural defence.

Terms

Specification gaming
Satisfying the letter of an objective against its intent.
Reward hacking
Exploiting a flaw in how reward is computed.
Misspecification
An objective that differs from what was intended.
Outcome check
Measuring the intended result directly, not through the reward.
A capable optimiser finds the best-paying behaviour the objective allows, intended or not; check the intended outcome, not the reward.

Sources

  1. Krakovna et al., “Specification gaming: the flip side of AI ingenuity”, DeepMind blog, 2020, and the accompanying list.
  2. Clark and Amodei, “Faulty Reward Functions in the Wild”, OpenAI blog, 2016 (the boat race).
  3. Skalse et al., “Defining and Characterizing Reward Hacking”, NeurIPS 2022.