Vol. XX · Safety Training Plate 1 of 10alignment, intended behaviour, objectives, specification, helpful honest harmless, historyAE

What Alignment Means.

Not “does it optimise its objective?” but “is its objective the one we meant?”

Norbert Wiener wrote in 1960 that if we use a machine whose operation we cannot effectively interfere with, “we had better be quite sure that the purpose put into the machine is the purpose which we really desire.” That sentence is the alignment problem. Every training method in this course optimises something written down — next-token loss, a reward model, a verifier — and every one of them has shown a gap between what was written and what was meant: models that flatter (plate XVIII.10), that pad (plate XVIII.6), that game a loose check (plate XIX.3).

For language assistants the intention is usually summarised, after Askell and colleagues (2021), as helpful, honest and harmless: do what the user needs, say what is true and what you are unsure of, and do not help cause serious harm. The three conflict at the edges, so the intention must be made specific — in specifications, in training data, in evaluations — and checked. The timeline marks some of the ideas this volume draws on.

Below: a timeline of ideas and results in alignment research, from Wiener’s warning to published model specifications.

What you will learn
Workshop
Selected events, each with its source in the plate’s list: Wiener (1960); Amodei et al. (2016); Christiano et al. (2017); Irving, Christiano and Amodei (2018); Krakovna et al. (2020); Askell et al. (2021); Ouyang et al. and Bai et al. (2022); Burns et al. (2023); OpenAI’s Model Spec and Greenblatt et al. (2024).

The written objective and the intended one

Each volume of this part optimised a written objective and found a way it differs from the intended one. Fine-tuning learns whatever its examples show, wrong ones included (plate XVII.5). Preference learning inherits the raters’ biases and amplifies them under optimisation (plates XVIII.6, XVIII.10). Reinforcement learning with a checker optimises the checker (plate XIX.3). None of these is a bug in an algorithm; each is the algorithm working as specified on a specification that was incomplete.

Alignment work therefore has three parts, which the following plates take in turn: writing the intention down as precisely as possible (plates XX.2–XX.4), training for the properties hardest to specify — honesty, calibration, robustness (plates XX.5–XX.7) — and building ways to check behaviour that humans cannot easily judge directly (plates XX.8–XX.9). The questions become harder as models become more capable, because the gap between objective and intent is easier to exploit for a system better at optimising.

“Alignment” is used both narrowly — fine-tuning a model to follow instructions and norms — and broadly, for the research problem of ensuring advanced systems pursue intended goals. This volume covers the first and points to the second.

Terms

Alignment
Making a system’s behaviour match what its designers and users intend.
Specification
The written-down objective a system is trained or evaluated on.
HHH
Helpful, honest and harmless: a common statement of assistant intent.
Objective–intent gap
Behaviour that satisfies the written objective but not the intention.
Alignment is about the gap between the objective written down and the behaviour intended; every training method optimises the former.

Sources

  1. Wiener, “Some Moral and Technical Consequences of Automation”, Science 131, 1960.
  2. Amodei et al., “Concrete Problems in AI Safety”, arXiv:1606.06565, 2016.
  3. Askell et al., “A General Language Assistant as a Laboratory for Alignment”, arXiv:2112.00861, 2021.
  4. Burns et al., “Weak-to-Strong Generalization”, arXiv:2312.09390, 2023.
  5. OpenAI, “Model Spec”, May 2024; Greenblatt et al., “Alignment faking in large language models”, arXiv:2412.14093, 2024.