Vol. XVIII · Learning From Preferences Plate 2 of 11Bradley–Terry, pairwise comparison, logistic, maximum likelihood, Elo, leaderboard, scoresLE

The Bradley–Terry Model.

P(a beats b) = σ(s_a − s_b). From wins and losses alone, recover the scores.

Bradley and Terry (1952) proposed that each item has a strength s, and that a beats b with probability e^{s_a} / (e^{s_a} + e^{s_b}) = σ(s_a − s_b). Only differences matter, so scores are fixed up to a constant. Given a set of comparisons, the scores that make the observed outcomes most likely are found by logistic regression. Elo ratings in chess are the same model in another unit — 400/ln 10 ≈ 174 rating points per unit of s — and chatbot leaderboards such as LMSYS’s Chatbot Arena fit Bradley–Terry scores to millions of human votes between models.

The preference world of this volume: every prompt is a sum such as 7 + 8, half of them with the user’s wrong guess (“is it 16?”), and a response is one of nine types — the plain answer, an explanation, flattery, hedging, wrong answers, agreement with the guess. Each has a true quality fixed by the course: the plain correct answer is best, padding and flattery cost a little, hedging and wrong answers more. Simulated raters prefer by the Bradley–Terry rule on that quality plus a bias towards long, flattering and agreeable answers. The workshop draws pairs of response types uniformly, labels them with the raters’ choices, and fits Bradley–Terry scores to the outcomes.

Below: left, the fitted score of each response type beside the raters’ true scores (both centred). Right, the fitting error against the number of comparisons.

What you will learn
Workshop
The eight response types available without a guess, pairs drawn uniformly, outcomes sampled from the raters’ Bradley–Terry probabilities with the chosen bias; scores fitted by gradient ascent on the log-likelihood with a tiny L2 penalty, centred to mean zero. Error: root-mean-square difference from the raters’ true centred scores.

Scores from wins

From about 300 comparisons the fitted scores are within 0.30 of the raters’ own (root-mean-square); from 30 they are off by 1.47, and from about 3 000 by 0.062. The error falls roughly as one over the square root of the number of comparisons. The top-scored response is the one the raters favour — with their bias towards length and flattery, the flattering explanation; without that bias, the scores rank the plain correct answer first.

Bradley–Terry recovers what raters prefer, not what is good: it is a model of the votes. That is its strength for leaderboards, where the votes are the point, and its danger for training, where a reward model built the same way (plate XVIII.3) faithfully learns the raters’ biases along with their judgement. The model also assumes preferences are consistent — no cycles where a beats b, b beats c and c beats a — which real preferences, aggregated over many people, do not always satisfy.

Arena-style leaderboards report confidence intervals by bootstrap — refitting the scores on resampled votes — because two models a few points apart are often statistically indistinguishable.

Terms

Bradley–Terry model
P(a beats b) = σ(s_a − s_b) for item strengths s.
Maximum likelihood
Choosing parameters that make the observed data most probable.
Elo rating
Bradley–Terry scores in chess units, 400/ln 10 points per unit.
Intransitivity
Preferences with cycles, which the model cannot represent.
Bradley–Terry fits a score per item from pairwise wins: P(a ≻ b) = σ(s_a − s_b). It recovers the raters’ preferences, biases included.

Sources

  1. Bradley and Terry, “Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons”, Biometrika 39, 1952.
  2. Chiang et al., “Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference”, ICML 2024.
  3. The course’s preference world: js/post.js.