Vol. XX · Safety Training Plate 8 of 10scalable oversight, debate, recursive reward modelling, weak-to-strong generalisation, verification, judgingAE

Oversight That Scales.

How does a weaker judge supervise a stronger model? By checking where an informed opponent points.

RLHF works while people can tell good outputs from bad. For outputs longer, more technical or more subtle than a rater can check, the training signal becomes noise or, worse, rewards what merely looks right. Scalable oversight is the research programme for this. Debate (Irving, Christiano and Amodei, 2018) has two models argue opposite sides before a judge, on the idea that pointing out a flaw is easier than finding it, so a judge can follow a debate it could not have produced. Recursive schemes split a task into parts small enough to judge. Weak-to-strong generalisation (Burns et al., 2023) studies whether a strong model trained on a weak supervisor’s labels can exceed the supervisor.

The workshop models the core argument of debate. A claim is a chain of k steps, one of which may be wrong. A judge who can afford to check a single step finds the error only if it happens to check the right one; in a debate, an opponent who can locate the error points to it, and the judge checks that step.

Below: the chance a wrong chain is caught, for a lone judge checking one random step and for a judge in a debate, against the length of the chain.

What you will learn
Workshop
A model of the argument, not of a trained system: a chain of k steps with one wrong step (when it is wrong at all, with probability q); a judge checks one step. Alone, it checks a random step. In a debate, the opponent points to the wrong step with probability s, and the judge checks that one.

Point to the flaw

For a 20-step claim, a judge who can check one step at random catches a wrong one 5.0% of the time. In a debate where the opponent finds the flaw 90% of the time and points to it, the judge catches it 90% of the time — 18.0 times as often — and that advantage grows with the length of the claim, because the judge’s work stays one step. If the opponent finds the flaw only half the time, the debate catches 50%: the scheme is only as strong as the debaters’ ability to find what is wrong.

Empirical work has found both promise and caveats. Experiments in which models debated reading-comprehension questions before judges who could not see the text found debate improved judges’ accuracy, more so with more capable debaters. Burns and colleagues found a strong model fine-tuned on a weak model’s labels recovers part of the gap between the weak and strong models’ abilities — generalising beyond its supervisor — but not all of it. None of these schemes is yet a guarantee; each is a way of stretching limited human judgement further, and each assumes the systems being overseen are not coordinating against the overseer.

The same asymmetry — checking is easier than producing — is why verifiable rewards work (plate XIX.3): a sum, a test or a proof checker is a judge that can supervise outputs far beyond what it could produce.

Terms

Scalable oversight
Supervising outputs that are too hard for the supervisor to judge directly.
Debate
Models arguing opposite sides before a judge, to expose flaws.
Weak-to-strong generalisation
A strong model trained on a weaker supervisor’s labels exceeding it.
Task decomposition
Splitting a task into parts small enough to judge.
Scalable oversight stretches limited judgement: an informed opponent pointing to the flaw lets a judge check one step instead of all.

Sources

  1. Irving, Christiano and Amodei, “AI safety via debate”, arXiv:1805.00899, 2018.
  2. Burns et al., “Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision”, arXiv:2312.09390, 2023.
  3. Bowman et al., “Measuring Progress on Scalable Oversight for Large Language Models”, arXiv:2211.03540, 2022.
  4. Khan et al., “Debating with More Persuasive LLMs Leads to More Truthful Answers”, ICML 2024.