Vol. XVIII · Learning From Preferences Plate 8 of 11IPO, SimPO, KTO, ORPO, DPO variants, length normalisation, reference-free, preference lossesLE

The DPO Family.

One derivation, many losses — and each moves the policy somewhere different.

DPO’s loss keeps decreasing as long as the chosen response’s log-ratio exceeds the rejected one’s by more; on preferences that are always the same, it pushes the margin to infinity and the policy far from the reference. IPO (Azar et al., 2023) instead regresses the margin to a fixed target, 1/(2β), so it stops. SimPO (Meng et al., 2024) removes the reference model and scores each response by its average log-probability per token, with a target margin γ, which also counteracts the tendency to prefer long responses. KTO (Ethayarajh et al., 2024) needs no pairs — each response is just labelled desirable or undesirable — and ORPO (Hong et al., 2024) adds an odds-ratio preference term to the ordinary SFT loss, so one stage does both.

The workshop trains the preference world’s policy with DPO, IPO and SimPO on the same 1 000 comparisons from biased raters, and compares where each one ends up: its true quality, its distance from the reference, and how long its answers are.

Below: left, the probability of each response type under the chosen loss beside the reference. Right, true quality, KL and mean length for all three losses.

What you will learn
Workshop
1 000 comparisons from raters of bias 1. Each loss trained for 300 steps of gradient descent on the policy’s six parameters at β = 0.3: DPO with the reward model’s L2 penalty on βθ; IPO with target margin 1/(2β); SimPO without the reference, lengths in words, γ = 0.5. Quantities computed exactly.

Small changes, different policies

On the same comparisons at β = 0.3, DPO lands at true quality 1.824 with KL 0.73 and answers of 12.1 words. IPO, whose squared loss stops pushing once the margin reaches its target, stays much closer to the reference — KL 0.10 — at quality 1.557. SimPO, normalising by length and using no reference, lands at quality 1.800 with answers of 8.0 words against the reference’s 7.7: dividing by length counters the raters’ taste for long answers, at the cost of anchoring to nothing.

None of these is “the” right loss; each encodes a judgement about the data. IPO suits preferences that are nearly deterministic, where DPO would over-fit; SimPO suits settings where length bias is the main problem and a reference model is a burden; KTO suits data that comes as single ratings rather than pairs, such as thumbs-up feedback from a product; ORPO suits a single-stage pipeline. Comparisons across papers are hard because each tunes its own β and data, and the differences are often smaller than the differences between datasets.

The family keeps growing — ODPO, cDPO, RSO, SPPO, and online variants that resample the current policy between rounds. The online variants close much of the gap to PPO that DPO’s offline nature opens (plate XVIII.7).

Terms

IPO
Identity preference optimisation: a squared loss towards a fixed margin.
SimPO
A reference-free loss on length-normalised log-probabilities with a target margin.
KTO
A loss on single responses labelled desirable or undesirable, from prospect theory.
ORPO
An odds-ratio preference term added to the SFT loss, with no reference model.
Each member of the DPO family changes one assumption — bounded margins, no reference, length normalisation, unpaired labels — and lands on a different policy.

Sources

  1. Azar et al., “A General Theoretical Paradigm to Understand Learning from Human Preferences”, AISTATS 2024 (IPO).
  2. Meng, Xia and Chen, “SimPO: Simple Preference Optimization with a Reference-Free Reward”, NeurIPS 2024.
  3. Ethayarajh et al., “KTO: Model Alignment as Prospect Theoretic Optimization”, ICML 2024.
  4. Hong, Lee and Thorne, “ORPO: Monolithic Preference Optimization without Reference Model”, EMNLP 2024.