Drills: Preferences.
Five questions at a time from the whole volume, until a preference pipeline reads like a chain of proxies.
This plate draws rounds of five from the questions and predictions of the ten plates before it, with the reasoning and source plate after each answer.
The volume’s thread is the proxy: comparisons stand in for judgement, a reward model for the comparisons, and the policy optimises the reward model — so every bias in the chain is learned, and optimisation amplifies it.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Explain why preference data is pairwise.
- Fit and read Bradley–Terry models.
- Relate RLHF, KL and DPO.
- Recognise over-optimisation and sycophancy.
What to carry forward
Comparisons cancel raters’ offsets (plate XVIII.1); Bradley–Terry turns them into scores (plate XVIII.2); a reward model generalises them, biases included (plate XVIII.3). RLHF maximises reward minus β KL with PPO (plate XVIII.4); the KL keeps the policy where the reward model is trusted (plate XVIII.5), and pushed too far, true quality falls (plate XVIII.6).
DPO reaches the same optimum without a reward model (plate XVIII.7), and its variants each change one assumption (plate XVIII.8). AI feedback with written principles can fix the biases it names (plate XVIII.9), and sycophancy is what optimisation makes of a preference for agreement (plate XVIII.10). Vol. XIX turns to rewards that can be checked.
Terms
- Bradley–Terry
- P(a ≻ b) = σ(s_a − s_b).
- KL penalty
- β KL(π ‖ π_ref).
- DPO
- Preferences fitted with the policy’s log-ratios.
- Goodhart
- A target stops being a measure.