Vol. XVII · Supervised Fine-Tuning Plate 9 of 10distillation, teacher, student, synthetic data, sequence-level distillation, soft targets, imitationLE

Distilling a Teacher.

Train a small model on a big model’s answers, and it inherits the big model’s skill — and its errors.

Knowledge distillation (Hinton, Vinyals and Dean, 2015) trains a small student to match a large teacher’s output distribution — its soft probabilities, which carry more information per example than a single right answer. For language models the common form is simpler: sequence-level distillation, which fine-tunes the student on text the teacher generated. Alpaca, Vicuna and Orca were built this way from stronger proprietary models, and many small open models are fine-tuned on a larger sibling’s outputs (plate XIX.11 on reasoning).

The workshop distils in the post-training world: a teacher answers all 70 non-held-out questions, right with a chosen probability and otherwise with a plausible wrong sum, and a student is fine-tuned on those 70 answers from the base. Its held-out accuracy shows how much of the teacher’s quality passes on.

Below: the student’s accuracy on held-out questions and on its training questions (against the true sums), for a teacher of the chosen accuracy.

What you will learn
Workshop
Live fine-tuning from the recorded base, 150 steps, loss on answers only, on the 70 non-held-out pairs labelled by a teacher of the chosen accuracy (a fixed share of its answers replaced by a plausible wrong sum). Accuracy on the 30 held-out pairs, and on the 70 training pairs against their true sums.

The student’s ceiling is the teacher

A perfect teacher gives the student 80% on held-out questions; a 90% teacher, 63%; a 70% teacher, 53%; a 50% teacher, 33%. On its own training questions the student reproduces what it was taught, wrong sums included — 84% right against the true sums with the 90% teacher, 31% with the 50% one — because to the loss a confident wrong answer looks exactly like a confident right one.

Distillation transfers a teacher’s behaviour very efficiently, which is why it is everywhere: a few hundred thousand teacher outputs can give a small model much of a large one’s style and some of its skill. But it transfers what the teacher does, not what is true; the student’s errors begin with the teacher’s, and a filter (plate XVII.4) is the only thing that can remove them. Imitation also copies surface without depth — studies of models distilled from larger ones found they matched the teacher’s tone more closely than its accuracy — and terms of use of some proprietary models forbid training competitors on their outputs.

Soft targets carry the teacher’s uncertainty: a teacher that gives 7 probability 0.6 and 8 probability 0.3 tells the student that 8 was a near miss — information a sampled hard label throws away.

Terms

Knowledge distillation
Training a student to match a teacher’s output distribution.
Sequence-level distillation
Fine-tuning a student on text a teacher generated.
Soft targets
The teacher’s full probability distribution, not just its top answer.
Student
The model being trained to imitate a teacher.
A distilled student inherits its teacher’s errors as faithfully as its skill; its ceiling is the teacher, unless something filters between them.

Sources

  1. Hinton, Vinyals and Dean, “Distilling the Knowledge in a Neural Network”, arXiv:1503.02531, 2015.
  2. Kim and Rush, “Sequence-Level Knowledge Distillation”, EMNLP 2016.
  3. Mukherjee et al., “Orca”, arXiv:2306.02707, 2023.
  4. Gudibande et al., “The False Promise of Imitating Proprietary LLMs”, arXiv:2305.15717, 2023.