Vol. XVII · Supervised Fine-Tuning Plate 5 of 10data quality, data quantity, label noise, curation, LIMA, memorisation, generalisationDE

Quality Over Quantity.

A few dozen right answers teach more than many answers of which some are wrong.

Pre-training improves with more data almost without limit (Vol. X). Fine-tuning behaves differently: the model already has its knowledge, and the examples mostly show it which behaviour is wanted. A wrong example is then not just a wasted example — it teaches the wrong behaviour as confidently as a right one teaches the right one. LIMA’s authors found 1 000 carefully curated examples competitive with far larger, noisier sets; many later recipes report the same in their own way, investing in filtering rather than volume.

The workshop fine-tunes the post-training world’s model on a chosen number of examples, a chosen share of which have wrong answers, and measures accuracy on held-out questions and on the taught questions themselves.

Below: held-out and taught-question accuracy as fine-tuning proceeds, for the chosen number of examples and share of wrong answers.

What you will learn

Contrast data scaling in pre-training and fine-tuning.
Show the effect of wrong answers in fine-tuning data.
Recognise memorisation of too few examples.
Explain why curation pays.

Workshop

Live fine-tuning from the recorded base, 150 steps, loss on answers only. The examples are the first n of the fixed shuffle (the first 30 human-written, the rest given correct answers here); the first ε·n of them have a wrong sum. Accuracy on the 30 held-out pairs and on the taught pairs against their true sums.

Right answers, enough of them

Thirty correct examples give 53% on held-out questions. Fifty examples of which twenty are wrong give 10%: more data, worse model, and on the taught questions it now reproduces the wrong sums (46% right). Ten correct examples are too few: the model learns all ten (100% of the taught questions) and almost nothing else (3% held out) — memorisation without the generalisation that comes from seeing the format across many different sums. Seventy correct examples give 80%.

Quantity still matters, then, but only up to the point where the behaviour has been shown in enough variety; beyond it, quality dominates. That is why fine-tuning sets are measured in thousands to hundreds of thousands of examples rather than trillions of tokens, and why so much effort goes into deciding which examples to keep. Real data has subtler errors than a wrong sum — a confident tone on a guess, an answer that is right but unhelpful — and they teach just as effectively.

Wrong labels hurt fine-tuning more than pre-training because each example is seen many times — 150 steps of batches of 16 over 50 examples is about 50 passes — so every error is rehearsed until it is learned.

Terms

Curation: Choosing and checking which examples to train on.
Label noise: Examples whose target response is wrong.
Memorisation: Learning the training examples without generalising beyond them.
Generalisation: Performing well on inputs not seen in training.

In fine-tuning a wrong example teaches the wrong behaviour; enough varied correct examples beat more examples with errors.

Sources

Zhou et al., “LIMA: Less Is More for Alignment”, NeurIPS 2023.
The course’s post-training world: js/post.js.