Vol. XVII · Supervised Fine-Tuning Plate 3 of 10loss masking, SFT loss, prompt tokens, answer tokens, label masking, completion-only loss, live trainingLE

Learning Only the Answers.

The prompt is something to read, not something to imitate. Mask it out of the loss.

A fine-tuning example is one sequence — prompt then answer — and next-token training would by default predict every token of it, the user’s words included. Loss masking sets the targets of the prompt positions to “ignore”, so the model still reads the prompt through attention but is trained only to produce the answer. How much it matters depends on how much of each example is prompt, and how unpredictable it is. When prompts are long — a system prompt, a document, chatty phrasing around a short question — most of the unmasked loss is about predicting the user.

The workshop fine-tunes the post-training world’s model with chatty prompts: up to six filler words such as “please”, “quick one” or “tell me” around each sum, as in “U:hi please 3+4? quick one|A:7|”. It trains once with the loss on the answers only and once on the whole sequence, for the same number of steps, and measures accuracy on the held-out questions.

Below: held-out accuracy as fine-tuning proceeds, with the loss on answers only and on the whole sequence. Readouts give the share of target tokens that belong to the answer.

What you will learn
Workshop
Live fine-tuning from the recorded base: 30 taught pairs, prompts with the chosen number of filler words, batches of 16, AdamW at 0.002, 150 steps. Masked: targets only on the answer and the end-of-turn mark. Unmasked: targets on every token of the example. Accuracy on the 30 held-out pairs, prompted with the same kind of filler.

Read the prompt, learn the answer

With four filler words around each question, the answer and its end mark are about 9% of the targets when nothing is masked. Training on everything for 150 steps leaves the model at 3% on held-out questions — it has spent its updates learning to predict “please” and “quick one” — while training on the answers alone reaches 33%. Even with no filler the two give 53% and 3%: the prompt’s digits are random, so predicting them is impossible, and the large, useless gradients of that impossible task drown the answers.

Real fine-tuning sets are full of long prompts — system messages, retrieved documents, previous turns of a conversation — so masking is the default in most SFT libraries. It is not a law: some recipes keep a small loss on prompts, and pre-training-style continued training deliberately learns everything. But a model meant to answer should be trained to answer, and the gradient should go where the behaviour is.

Multi-turn conversations are usually trained with every assistant turn unmasked and every user and system turn masked, so one example teaches several replies in context.

Terms

Loss masking
Excluding some positions from the loss, here the prompt.
Target
The token a position is trained to predict.
Completion-only loss
A loss computed only on the response tokens.
Filler
Words in a prompt that carry no information about the answer.
Mask the prompt so the loss — and the gradient — goes to the answer; it matters most when prompts are long and answers short.

Sources

  1. Ouyang et al., NeurIPS 2022 (loss on demonstrations).
  2. Hugging Face TRL documentation, “completion-only” training.
  3. The course’s post-training world: js/post.js.