Vol. XVIII · Learning From Preferences Plate 1 of 11preference data, pairwise comparison, Likert scale, rater agreement, calibration, annotationLE

Comparisons, Not Labels.

Two raters rarely give the same score. They much more often pick the same winner.

For tasks with one right answer, supervision is a label. For “write a helpful reply”, there is no single right answer, and people must judge. One option is a score — rate this response from 1 to 7 — but raters differ in how they use a scale: one is generous, another harsh, a third uses only the middle. Their scores mix the response’s quality with their own calibration. The other option is a comparison — which of these two is better? — in which each rater’s offset cancels, because both responses are judged on the same personal scale at the same moment.

InstructGPT, Anthropic’s helpful-and-harmless data and most preference datasets since collect comparisons for this reason. The workshop simulates two raters, each with a random offset and scale, judging pairs of responses of known quality with some noise, both by scoring each on a 1–7 scale and by choosing between them; it counts how often the two raters agree.

Below: how often two raters agree on which of two responses is better, when the winner is read from their 1–7 scores and when they choose directly, against the noise in their judgements.

What you will learn

Explain why open-ended tasks need human judgement.
Explain how rater calibration corrupts absolute scores.
Show that comparisons cancel each rater’s offset.
Explain why preference datasets are pairwise.

Workshop

Two raters with random offsets o_r and scales s_r; responses drawn from eight types of known quality u; Gaussian noise of the chosen size. Scores are rounded and clipped to 1–7, and a pair’s winner is read from them (equal scores are ties). Comparisons choose directly. 400 pairs.

Offsets cancel in a comparison

With these two raters and moderate noise, a winner read from their 1–7 scores agrees between them 61% of the time and ties 32% of the time; asked to choose directly, they agree 74%. They give the same score to the same response only 24% of the time, because one rater’s whole scale sits above the other’s. With quieter raters the comparisons agree 79% and the scores 62%.

A comparison asks a question people answer more consistently, and it produces exactly the data the next plates need: pairs of a preferred and a rejected response. Real preference collection adds refinements — ranking four to nine responses at once and splitting the ranking into pairs, as InstructGPT did; a “how much better” strength; ties allowed — but the unit is the comparison. Its limit is that it says which is better, not whether either is good.

Comparisons inherit the raters’ shared biases even as they cancel individual offsets: if every rater prefers longer answers, every comparison says so. That bias reaches the reward model (plate XVIII.3) intact.

Terms

Comparison: A judgement of which of two responses is better.
Likert scale: An absolute rating on a fixed scale, such as 1 to 7.
Calibration: How a rater maps quality to scores: offset and scale.
Inter-rater agreement: How often two raters give the same judgement.

A comparison cancels each rater’s offset and scale, so raters agree on comparisons far more than on scores.

Sources

Ouyang et al., NeurIPS 2022 (pairwise rankings of model outputs).
Bai et al., “Training a Helpful and Harmless Assistant with RLHF”, arXiv:2204.05862, 2022.
The course’s preference world: js/post.js.