Vol. XVII · Supervised Fine-Tuning Plate 4 of 10instruction data, human demonstrations, self-instruct, synthetic data, filtering, verification, data qualityDE

Instruction Data.

Written by people, written by models, or written by models and checked — three sources, three very different results.

The first instruction-tuned models were trained on demonstrations written by paid annotators, which are good and expensive. Self-Instruct (Wang et al., 2023) showed a model could write its own: from 175 seed tasks, a model generated new instructions and answers, and Stanford’s Alpaca used the same recipe with a stronger model to produce 52 000 demonstrations for a few hundred dollars. Since then most instruction data has been synthetic — written by models — and the question is how to keep its errors out.

The world of Vols. XVII–XIX: a 27 000-parameter character Transformer pre-trained, by this page’s own engine, on arithmetic worksheets (“3+4=7. 9+1=10. …”) until it completes every one of the hundred sums; fine-tuning teaches it a chat format, “U:3+4?|A:7|”. Of the hundred pairs of digits, 30 have written answers, 40 are prompts without answers, and 30 are held out to measure what the model can do on questions it was never shown in chat form. Here a teacher, fine-tuned on the 30 human-written examples, answers the 40 unanswered prompts. A student is then fine-tuned from the base on the human examples alone, on those plus every model-written answer, or on those plus only the model-written answers that a checker — here, simply adding the digits — verifies.

Below: the student’s held-out accuracy as fine-tuning proceeds, for the chosen data source; readouts give how many examples it had and how many of the teacher’s answers were right.

What you will learn
Workshop
Live fine-tuning from the recorded base, 150 steps, loss on answers only. Teacher: the model fine-tuned on the 30 human examples (as in plate XVII.1), answering the 40 unanswered prompts greedily. Model-written: all 40 of its answers added, right or wrong. Verified: only those whose sum is correct. Accuracy on the 30 held-out pairs.

Synthetic data needs a filter

On its 30 human examples alone the student reaches 53% on the held-out questions. The teacher gets 60% of the 40 unanswered prompts right; adding all 40 of its answers gives the student more data and 37% — worse than before, because it now learns the teacher’s wrong sums with the same confidence as its right ones. Keeping only the 24 answers the checker verifies gives 54 examples and 73%: better than the human data alone.

That is the pattern of modern instruction data. Models write most of it — new prompts, answers, rewrites, multi-turn dialogues — and the effort goes into filtering: verifiable answers checked by execution or by exact match (Vol. XIX), the rest scored by reward models or judge models (plate XXIV.5), deduplicated against benchmarks (plate X.4) and against itself. The checker here is perfect, which real ones rarely are; a filter that lets errors through, or rejects good answers in a biased way, passes its own mistakes to the student.

A model trained on its own outputs without new information can only reinforce what it already does; it is the filter — a verifier, a stronger judge, a human — that adds information and makes the loop improve rather than drift (plate IX.7 on model collapse).

Terms

Demonstration
An example prompt with the response a model should give.
Self-Instruct
Using a model to generate instruction-tuning data from a few seed tasks.
Synthetic data
Training data written by models.
Verifier
A check that decides whether a response is correct.
Model-written data carries the model’s errors; a verifier that keeps only correct answers turns it from harmful to helpful.

Sources

  1. Wang et al., “Self-Instruct”, ACL 2023.
  2. Taori et al., Stanford Alpaca, 2023.
  3. The course’s post-training world: js/post.js.