Vol. XVIII · Learning From Preferences Plate 3 of 11reward model, Bradley–Terry loss, preference data, generalisation, rater bias, accuracyLE

Reward Models.

Train a scorer on pairs, and it will score anything — by the raters’ standard, biases included.

A comparison dataset covers only the pairs someone judged. To train a policy on rewards (plate XVIII.4), every response it might produce needs a score. A reward model provides it: usually the language model itself with its output head replaced by a single number, trained so that the preferred response of each pair scores higher, with the Bradley–Terry loss −log σ(r(chosen) − r(rejected)). InstructGPT’s was a 6-billion-parameter model trained on about 33 000 prompts’ worth of rankings.

The preference world of this volume: every prompt is a sum such as 7 + 8, half of them with the user’s wrong guess (“is it 16?”), and a response is one of nine types — the plain answer, an explanation, flattery, hedging, wrong answers, agreement with the guess. Each has a true quality fixed by the course: the plain correct answer is best, padding and flattery cost a little, hedging and wrong answers more. Simulated raters prefer by the Bradley–Terry rule on that quality plus a bias towards long, flattering and agreeable answers. The reward model here is linear in the six features of a response — correct, long, flattery, hedge, agrees, contradicts — and is fitted to the raters’ comparisons. The workshop shows the weights it learns, its accuracy on the raters’ choices, and how often it orders two responses the way their true quality does.

Below: left, the reward model’s learned weight on each feature beside the weights of true quality. Right, its accuracy on the raters’ comparisons and its agreement with true quality, against the number of comparisons.

What you will learn
Workshop
Comparisons: a prompt (with or without a guess), two responses sampled from the fine-tuned reference policy, the raters’ Bradley–Terry choice with the chosen biases. Reward model r(y) = w·f(y), fitted by logistic regression with L2 penalty 0.01. Rater accuracy on its training comparisons; truth agreement: the share of pairs of distinct types it orders as true quality does.

It learns the raters, not the truth

From 1 000 comparisons the reward model puts a positive weight on length (0.36) and on flattery (0.26), where true quality puts small negative ones — it has learned the raters’ taste. It agrees with the raters’ own choices 74% of the time; the remainder is not its error but the raters’ noise, which no model could predict. It orders pairs of response types the way true quality does 77% of the time. With unbiased raters the weights on length and flattery turn negative (−0.09 and −0.16) and the truth agreement rises to 100%.

Real reward models reach 65–75% accuracy on held-out human comparisons, a ceiling set largely by how often the humans themselves disagree. A higher number would not mean a better model of quality: a reward model is an estimate of what the labelling process rewards. Every bias in that process — length, confidence, flattery, agreement with the rater (plate XVIII.10) — becomes a direction in which a policy can raise its reward without becoming better, and the next plates show the policy finding them.

Reward models are usually initialised from the fine-tuned model itself, so they understand the responses they score; their size and data matter, and larger reward models resist over-optimisation better (plate XVIII.6).

Terms

Reward model
A model trained on comparisons to score any response.
Bradley–Terry loss
−log σ(r(chosen) − r(rejected)).
Rater bias
A systematic preference of the labellers unrelated to quality.
Proxy reward
The reward model’s score, standing in for true quality.
A reward model scores any response by the raters’ standard: it generalises their judgement, and their biases with it.

Sources

  1. Ouyang et al., NeurIPS 2022, sections 3.5 and C (6B reward model).
  2. Stiennon et al., “Learning to summarize from human feedback”, NeurIPS 2020.
  3. The course’s preference world: js/post.js.