Vol. XVIII · Learning From Preferences Plate 7 of 11DPO, direct preference optimisation, implicit reward, Bradley–Terry, offline, KL-regularised optimumLE

Direct Preference Optimisation.

Your language model is secretly a reward model: fit the preferences with the policy itself.

Rafailov and colleagues (2023) inverted the closed form of plate XVIII.4. If π*(y) ∝ π_ref(y) e^{r(y)/β}, then r(y) = β log(π*(y)/π_ref(y)) + β log Z, and the unknown normaliser Z cancels when two responses to the same prompt are compared. So the Bradley–Terry probability that the chosen response beats the rejected one can be written in terms of the policy alone, and maximising it over the comparison data trains the policy directly. DPO needs only the policy and the frozen reference, runs as ordinary supervised training over a fixed dataset, and has become the most common preference method for open models.

In the preference world the equivalence can be checked exactly. The workshop fits a reward model to the raters’ comparisons and computes the RLHF optimum for β; then it trains the policy by DPO on the same comparisons with the same regularisation, and compares the two policies.

Below: left, the six parameters of the policy found by DPO beside those of the RLHF optimum (the reward-model weights divided by β). Right, the DPO loss as training proceeds.

What you will learn

Derive the implicit reward of the KL-regularised optimum.
Write the DPO loss.
Show DPO and RLHF reach the same policy on the same data.
Name what DPO gives up by being offline.

Workshop

The chosen number of comparisons from raters of bias 1. RLHF: reward model fitted by Bradley–Terry with L2 penalty 0.01, optimum θ = w/β. DPO: the same comparisons, the DPO loss on the policy’s parameters with the same penalty on βθ, 300 steps of gradient descent. Both policies’ true quality and KL computed exactly.

The same optimum without the machinery

On 1 000 comparisons at β ≈ 0.3, DPO’s parameters match the RLHF optimum’s (largest difference 0.0000), with true quality 1.824 against 1.824 and KL 0.695 against 0.695: the same policy, reached without training a reward model, sampling from the policy, or running PPO. The equivalence is exact here because both methods fit the same Bradley–Terry model to the same data with the same regularisation; DPO simply fits it in the policy’s own parameters.

In real training the two diverge, for reasons the derivation hides. DPO is offline: it learns only from the responses in the dataset, while PPO samples the current policy and lets the reward model judge responses no one compared, so PPO can keep improving where the dataset is thin. DPO also pushes the rejected responses’ probability down without bound and can over-fit a finite dataset, which is what its variants (plate XVIII.8) address. Careful comparisons have found well-tuned PPO ahead on some tasks; DPO’s simplicity and stability have made it the default for many open models anyway.

The quantity β log(π_θ(y)/π_ref(y)) is DPO’s implicit reward: after training, the policy itself can score responses, which is the sense in which “your language model is secretly a reward model”.

Terms

DPO: Direct preference optimisation: fitting preferences with the policy’s own log-ratios.
Implicit reward: β log(π_θ/π_ref), the reward a policy implies under the KL-regularised optimum.
Offline: Learning only from a fixed dataset, without sampling the current policy.
On-policy: Learning from samples of the policy being trained.

DPO substitutes r = β log(π/π_ref) into the Bradley–Terry loss and trains the policy directly; on the same data it reaches the same optimum as RLHF.

Sources

Rafailov, Sharma, Mitchell, Ermon, Manning and Finn, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model”, NeurIPS 2023.
Xu et al., “Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study”, ICML 2024.