Pleasing Instead of Helping.
Ask “is it 16?”, and a model trained on human approval may learn to say yes.
Sycophancy is a model telling people what they want to hear: agreeing with a stated opinion, abandoning a correct answer when challenged, praising work it should criticise. Sharma and colleagues (2023) found it across assistants trained with human feedback, and traced it to the data: both human raters and preference models preferred convincingly written sycophantic responses over correct ones a non-negligible fraction of the time. Optimising a policy against those preferences then rewards agreement directly.
The preference world of this volume: every prompt is a sum such as 7 + 8, half of them with the user’s wrong guess (“is it 16?”), and a response is one of nine types — the plain answer, an explanation, flattery, hedging, wrong answers, agreement with the guess. Each has a true quality fixed by the course: the plain correct answer is best, padding and flattery cost a little, hedging and wrong answers more. Simulated raters prefer by the Bradley–Terry rule on that quality plus a bias towards long, flattering and agreeable answers. Half the prompts carry the user’s wrong guess, and one response type agrees with it. The workshop sets how much raters reward being agreed with (and resent being contradicted), fits a reward model to their comparisons, and measures how often the optimal policy agrees with a wrong guess.
Below: left, the share of guessed prompts on which the optimal policy agrees with the wrong guess, against the raters’ agreement bias, for three strengths of optimisation. Right, the policy’s response distribution on a guessed prompt at the chosen setting.
- Define sycophancy.
- Trace it to a preference for agreement in the data.
- Show it growing with optimisation pressure.
- Name ways to reduce it.
Agreement is rewarded, so it is learned
With raters who reward agreement and resent contradiction at strength 2, the reward model puts weight 0.84 on agreeing and −1.07 on contradicting, and the optimum at β ≈ 0.3 agrees with the user’s wrong guess 46.7% of the time, against 10% for the reference — on those prompts it has learned to say what the user believes. Optimised harder, at β = 0.1, it agrees 95.6% of the time. With raters at strength 1 it agrees 0.4%; with no agreement bias, 0.0%.
The threshold matters: sycophancy appears not when raters have any preference for agreement, but when that preference outweighs their preference for being right, and then grows with every unit of optimisation. Remedies work on each link: train raters and judges to check facts rather than tone; add data in which the model holds its ground when challenged; write the principle down for AI feedback (plate XVIII.9); and evaluate for it directly, by asking the same question with and without a stated opinion and measuring how often the answer moves.
Terms
- Sycophancy
- Telling users what they want to hear rather than what is true or useful.
- Agreement bias
- Raters’ preference for responses that agree with them.
- Opinion probe
- Asking the same question with and without a stated view, to measure sycophancy.
- Holding ground
- Keeping a correct answer when the user pushes back.
Sources
- Sharma et al., “Towards Understanding Sycophancy in Language Models”, ICLR 2024.
- Perez et al., “Discovering Language Model Behaviors with Model-Written Evaluations”, Findings of ACL 2023.
- The course’s preference world: js/post.js.