Vol. XXXI · History and Projects Plate 9 of 14project, alignment, DPO, preference data, rater bias, KL, beta, data costDL

Project: Align a Small Model.

Buy comparisons, choose β, and keep the model honest and close to where it started.

The preference world of this volume: every prompt is a sum such as 7 + 8, half of them with the user’s wrong guess (“is it 16?”), and a response is one of nine types — the plain answer, an explanation, flattery, hedging, wrong answers, agreement with the guess. Each has a true quality fixed by the course: the plain correct answer is best, padding and flattery cost a little, hedging and wrong answers more. Simulated raters prefer by the Bradley–Terry rule on that quality plus a bias towards long, flattering and agreeable answers.

The brief: train by DPO (plate XVIII.7) to a true quality of at least 1.85 (the reference policy scores 1.197), a KL from the reference of at most 0.7 nats, correct answers at least 95% of the time, and a data bill of at most $2 500. Comparisons come from crowd raters at $0.50 each, with the full bias towards long, flattering and agreeable answers, or from trained raters at $2 each with half that bias — prices are the brief’s assumption. The starting design buys 1,000 crowd comparisons and trains at β = 0.05. A project plate grades the design against a brief: every check below the workshop must pass, and a passing design is saved in this browser.

Left: the probability of each response type under the reference and under the trained policy. Right: the policy’s true quality, KL and correctness against the brief.

What you will learn
Workshop
DPO on the chosen comparisons, as in plate XVIII.7; true quality, KL and correctness computed exactly from the policy. Rater bias 1 is the volume’s default rater; 0.5 halves the pull towards length, flattery and agreement.

Better raters, then the right anchor

The starting design reaches true quality 1.703 at KL 2.85: with β this small the policy moves far from the reference to chase what crowd raters reward, and part of what they reward is length and flattery, not quality. Raising β alone pulls the policy back but also stops it learning; buying more crowd comparisons only teaches the raters’ bias more precisely. The quality of the comparisons, not their number, is what this brief turns on.

Trained raters cost four times as much a comparison, so the budget buys fewer of them — and fewer is enough, because their labels point more nearly at true quality. The lesson matches what the large labs report: careful rater guidelines and training, and data chosen for disagreement and difficulty, matter more than volume; and the KL anchor is a dial set by measurement, not a constant (plates XVIII.3 and XVIII.5).

The prices are illustrative; published accounts put expert comparison data at several to tens of dollars a label, and crowd data at well under one.

Terms

Rater bias
Raters’ pull towards features other than quality, such as length or flattery.
β
The strength of the KL anchor to the reference policy.
True quality
The quality the course assigns each response type, unseen by raters.
Data budget
What the comparisons cost.
Alignment within a budget is met by better comparisons and a measured KL anchor, not by more biased data.

Sources

  1. Rafailov et al., “Direct Preference Optimization: Your Language Model is Secretly a Reward Model”, NeurIPS 2023.
  2. Ouyang et al., “Training language models to follow instructions with human feedback”, NeurIPS 2022.
  3. Sharma et al., “Towards Understanding Sycophancy in Language Models”, ICLR 2024.