Vol. XXI · Evaluation Plate 6 of 10arena, Chatbot Arena, Elo, Bradley–Terry, pairwise votes, leaderboard, bootstrap, confidence intervalsEL

Head-to-Head.

Two anonymous answers, one vote. Millions of votes, one leaderboard — with error bars.

LMSYS’s Chatbot Arena (Chiang et al., 2024) let anyone ask a question, see two anonymous models’ answers, and vote for the better. With millions of votes it became one of the most watched leaderboards. Its ratings were first computed with online Elo, the chess system that nudges both players’ ratings after each game; it later switched to fitting the Bradley–Terry model to all votes at once (plate XVIII.2), which uses every vote fully and does not depend on the order they arrived in, and to reporting bootstrap confidence intervals.

The workshop simulates an arena of eight models with known true ratings 30 Elo points apart, draws the chosen number of votes between random pairs, and estimates the ratings both ways.

Below: left, each model’s true rating, its online Elo estimate and its Bradley–Terry estimate with a 90% bootstrap interval. Right, the error of each method against the number of votes.

What you will learn
Workshop
Eight models with true ratings 0, 30, …, 210 (shown centred on 1 000); each vote compares a random pair and is won with the Bradley–Terry probability. Online Elo with K-factor 16, starting at 1 000. Bradley–Terry fitted to all votes; 90% intervals from 30 bootstrap resamples. Errors: root-mean-square from the true centred ratings.

Fit all the votes at once

From about 1 800 votes, online Elo misplaces the models by 33 Elo points on average, and Bradley–Terry fitted to the same votes by 7. Online Elo is noisier because each vote moves the ratings by a fixed step and recent votes count most; fitting all votes together uses each one fully. A 90% interval on one model is about 47 points wide here, against true gaps of 30 — neighbouring models overlap and should not be ranked against each other. With 100 votes Bradley–Terry is off by 36; with 10 000, by 5.

An arena rating measures which answers voters prefer, on the questions they chose to ask, in the style they reward — length and formatting among them (plate XXI.5). It is valuable precisely because it is not a fixed benchmark and is hard to train against directly. It has its own failure modes: the population of questions shifts, models can be tested privately before release and the best variant reported, and the votes of a self-selected public are not the judgement of experts on hard questions. Like every number in this volume, it needs its interval and its context.

Elo’s K-factor trades responsiveness against noise: a large K follows changes quickly and jitters; a small K is steady and slow. Fitting all votes at once removes the trade-off for a fixed set of models.

Terms

Arena
An evaluation in which people vote between anonymous answers from two models.
Elo rating
Ratings updated after each game by K times the surprise of the result.
Bradley–Terry fit
Ratings chosen to make all observed votes most likely.
Bootstrap interval
An interval from refitting on resampled votes.
Arena ratings fitted by Bradley–Terry on all votes beat online Elo; read them with their intervals and as measures of preference.

Sources

  1. Chiang et al., “Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference”, ICML 2024.
  2. Elo, The Rating of Chessplayers, Past and Present, 1978.
  3. Singh et al., “The Leaderboard Illusion”, arXiv:2504.20879, 2025 (private testing and selective disclosure on arena leaderboards).