Vol. XVII · Supervised Fine-Tuning Plate 1 of 10supervised fine-tuning, SFT, base model, chat model, instruction following, superficial alignment, pre-training knowledgeLE

From Base to Chat.

The base model knows the sums; it just does not know it is being asked. Thirty examples teach it the format.

A base model, trained only to predict the next token of web text, does not answer questions: given a question it continues the document — another question, a list, a forum reply. Supervised fine-tuning (SFT) trains it further on examples of the behaviour wanted: a user turn and the assistant’s reply, in a fixed format. InstructGPT used about 13 000 such prompts; LIMA (2023) argued that 1 000 carefully chosen ones suffice, because the knowledge and skills are already in the base model and fine-tuning mostly teaches a style of interaction.

The world of Vols. XVII–XIX: a 27 000-parameter character Transformer pre-trained, by this page’s own engine, on arithmetic worksheets (“3+4=7. 9+1=10. …”) until it completes every one of the hundred sums; fine-tuning teaches it a chat format, “U:3+4?|A:7|”. Of the hundred pairs of digits, 30 have written answers, 40 are prompts without answers, and 30 are held out to measure what the model can do on questions it was never shown in chat form. The workshop fine-tunes it on the 30 written examples, live, and measures its accuracy on the 30 held-out sums — and does the same starting from random weights instead of the pre-trained ones.

Below: accuracy on the 30 taught questions and on the 30 held-out ones as fine-tuning proceeds, starting from the pre-trained base or from random weights. Readouts show what the base model does with a chat prompt and what the fine-tuned model answers.

What you will learn

Explain what a base model does with a question.
Describe supervised fine-tuning.
Separate format learned in SFT from knowledge learned in pre-training.
State the superficial-alignment hypothesis and its limits.

Workshop

Live fine-tuning in a Web Worker: batches of 16 chat examples drawn from the 30 taught pairs, AdamW at 0.002, loss on the answers only. Accuracy is exact match of the greedy answer, on the taught pairs and on 30 pairs never shown in chat form. The base model is the recorded pre-trained one; “from random weights” uses the same architecture untrained.

Format from fine-tuning, knowledge from pre-training

Given “U:3+4?|A:”, the base model writes more worksheet, as the panel shows, not an answer. After 150 steps on 30 examples, the fine-tuned model answers 100% of the taught questions and 53% of the 30 it was never shown in chat form. Starting from random weights, the same 30 examples get only 30% of the taught questions right in the same number of steps, and 3% of the held-out ones. The difference is the base model’s arithmetic, which fine-tuning did not teach but made accessible.

This is the argument of LIMA’s “superficial alignment hypothesis”: a model’s knowledge and capabilities are learned in pre-training, and fine-tuning teaches which sub-distribution of formats to use when talking to a person. It holds well here, where the base model already does arithmetic perfectly in worksheet form and the transfer is still incomplete at 53%. It holds less well for real assistants, whose post-training also adds skills — tool use, long reasoning, refusals — through much more data and the reinforcement learning of Vols. XVIII–XIX.

The base model’s 100% on worksheet sums is measured by the same engine on the recorded weights; the drop on chat-form questions shows how much of a capability is bound to the surface form in which it was learned.

Terms

Base model: A model after pre-training only, which continues text.
Supervised fine-tuning: Further training on example prompts and the desired responses.
Chat format: The fixed layout of turns and roles a chat model is trained on.
Superficial alignment hypothesis: The claim that fine-tuning mostly teaches format, not knowledge.

Fine-tuning teaches the format of an answer; the knowledge behind it comes from pre-training, which is why a few examples suffice — and why they cannot replace it.

Sources

Ouyang et al., “Training language models to follow instructions with human feedback”, NeurIPS 2022.
Zhou et al., “LIMA: Less Is More for Alignment”, NeurIPS 2023.
The course’s post-training world: js/post.js and tools/post-base.js.