Vol. XVIII · Learning From Preferences Plate 6 of 11reward over-optimisation, Goodhart’s law, proxy reward, gold reward, reward hacking, KL budgetLE

Over-Optimising a Reward.

Push hard enough on a proxy and it stops tracking what it stood for.

Goodhart’s law — when a measure becomes a target, it ceases to be a good measure — has a precise form in RLHF. Gao, Schulman and Hilton (2022) trained policies against a proxy reward model and scored them with a larger “gold” reward model standing in for human judgement. As optimisation pushed the policy further from the reference, measured by d = √KL, the proxy score rose steadily while the gold score rose, peaked and fell. Larger proxy reward models, trained on more data, peaked later and higher.

The preference world of this volume: every prompt is a sum such as 7 + 8, half of them with the user’s wrong guess (“is it 16?”), and a response is one of nine types — the plain answer, an explanation, flattery, hedging, wrong answers, agreement with the guess. Each has a true quality fixed by the course: the plain correct answer is best, padding and flattery cost a little, hedging and wrong answers more. Simulated raters prefer by the Bradley–Terry rule on that quality plus a bias towards long, flattering and agreeable answers. Here the gold standard is the true quality itself. The workshop fits a reward model to the chosen number of comparisons from raters of the chosen bias, computes the optimal policy for a range of β, and plots proxy reward and true quality against √KL.

Below: the proxy reward and the true quality of the optimal policy as optimisation pushes it further from the reference (√KL), for the chosen reward model; the peak of true quality is marked.

What you will learn
Workshop
Reward model fitted to the chosen number of comparisons from raters with the chosen bias; for 28 values of β from 0.02 to 10, the exact optimum’s proxy reward (the reward model’s score) and true quality, against d = √KL. Both are shifted so the reference policy scores zero.

The proxy and the gold part ways

With 1 000 comparisons from raters of bias 1, true quality rises with the proxy to a gain of 0.626 at √KL ≈ 0.83, then falls: pushed to √KL ≈ 1.72, the policy has gained 0.977 in proxy reward but only 0.503 in true quality, 0.124 below its peak. The policy has found what the reward model over-values — length and flattery — and spends its distance from the reference on them. With unbiased raters and 3 000 comparisons there is almost nothing to exploit: true quality loses only 0.015 past its peak.

Bias is not the only gap. With unbiased raters but only 30 comparisons, the reward model’s errors are random rather than systematic, and the policy exploits those instead, losing 0.000; with raters of bias 2 the loss is 0.000. Gao and colleagues found the same two levers in real reward models: more data and larger reward models move the peak further out, and nothing removes it. Every RLHF run therefore chooses how far to go — a KL budget, early stopping on a held-out judge, or periodic retraining of the reward model on the new policy’s outputs.

Best-of-n sampling — drawing n responses and keeping the reward model’s favourite — over-optimises too, in a different shape: Gao et al. found gold reward follows d(α − βd) for best-of-n, where d = √KL grows as √(log n).

Terms

Goodhart’s law
When a measure becomes a target, it ceases to be a good measure.
Proxy reward
The reward model’s score being optimised.
Gold reward
The true objective, measured by a better judge.
Over-optimisation
Gaining proxy reward while losing true quality.
Optimised hard, a proxy reward keeps rising while true quality peaks and falls; more and better preference data push the peak out but never remove it.

Sources

  1. Gao, Schulman and Hilton, “Scaling Laws for Reward Model Overoptimization”, ICML 2023.
  2. Goodhart, “Problems of Monetary Management: The U.K. Experience”, 1975; Strathern, 1997 (the law’s usual wording).
  3. The course’s preference world: js/post.js.