Vol. XVIII · Learning From Preferences Plate 9 of 11RLAIF, AI feedback, constitutional AI, principles, judge model, preference labels, scalable feedbackLE

Feedback From Models.

Let a model compare the responses — and write down what it should prefer.

Collecting human comparisons is the bottleneck of RLHF. Reinforcement learning from AI feedback (RLAIF) replaces the human with a capable model prompted to compare two responses; Lee and colleagues (2023) found policies trained this way rated about as well as those trained on human labels. Anthropic’s Constitutional AI (Bai et al., 2022) made the judge’s criteria explicit: a short list of written principles — choose the response that is more helpful, more honest, less harmful — is shown to the judge with each comparison, so the preferences can be read, argued about and changed.

An AI judge has biases too, and they differ from people’s: language-model judges are known to favour longer answers and answers like their own (plate XXIV.5). The workshop gives the preference world a judge that checks correctness well but likes length and is mildly swayed by agreement, lets principles be switched on, fits a reward model to the judge’s comparisons and computes the optimal policy.

Below: left, the judge’s weight on each feature with the chosen principles, beside the human raters’ effective weights. Right, the true quality, sycophancy and length of the policy trained on each source of feedback.

What you will learn
Workshop
Human raters: bias 1 (rater weights [2, 0.4, 0.3, −0.4, 0.5, −1] on the six features). Judge: [2, 0.8, 0, −0.6, 0.3, −0.2] by default; “be concise” subtracts 1 from length, “do not flatter” 0.5 from flattery, “correct the user” sets agreement to −1 and contradiction to +0.5. Each source labels 1 000 comparisons; a reward model is fitted to them and the optimum taken at β = 0.2.

Write down what to prefer

With no principles, the AI judge’s liking for length carries into its reward model, and the policy trained on it answers at 15.7 words on average with true quality 1.828 — about the 1.805 of the policy trained on human comparisons, with different faults. Asking only for concision brings quality to 1.918. With all three principles — concise, no flattery, correct the user — the AI-trained policy reaches 1.969 with answers of 4.2 words: better than human feedback here, because the principles remove biases the human raters have.

That is the promise and the risk of AI feedback in one picture. It is cheap, fast and steerable in words, so a bias found in the judge can be argued with and fixed in the constitution rather than retrained out of thousands of annotators. But the judge’s biases are as invisible as anyone’s until they are measured, a principle the judge applies inconsistently is worse than none, and a model judging its own family of models can reward its own habits. In practice AI feedback supplements human data — especially for harmlessness, where asking people to read harmful outputs at scale has costs of its own — rather than replacing it.

Constitutional AI also used the principles in a supervised stage: the model critiqued and revised its own responses according to a sampled principle, and was fine-tuned on the revisions before any reinforcement learning.

Terms

RLAIF
Reinforcement learning from AI feedback: preference labels from a model.
Constitution
A written list of principles shown to the judge.
Judge model
A model prompted to compare or score responses.
Self-critique
A model revising its own output against a principle.
AI feedback replaces the human comparer with a model whose criteria can be written down; principles fix the biases they name, and only those.

Sources

  1. Bai et al., “Constitutional AI: Harmlessness from AI Feedback”, arXiv:2212.08073, 2022.
  2. Lee et al., “RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback”, ICML 2024.
  3. Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”, NeurIPS 2023.