The Objective.
One number to minimise — the mean surprise over trillions of positions — and most of it comes from a minority of hard tokens.
Everything in a pretraining run serves a single objective: the cross-entropy of the model’s next-token predictions, averaged over every position of every training document. Each sequence in a batch contributes one loss per position, all computed in one forward pass thanks to the causal mask (plate IV.4); the batch’s average is backpropagated; the optimiser steps. Repeat for weeks. There is no separate objective for grammar, facts or reasoning — those are learned only insofar as they lower this average.
An average conceals its parts. Many tokens of real text are almost determined by what comes before — the second half of a split word, a closing bracket, “of” after “one” — and cost nearly nothing. Others, such as the first content word of a new sentence or a number, are genuinely uncertain. The workshop measures the spread of per-token losses of the course model micro on held-out text.
Below: the distribution of micro’s per-token losses on held-out course text, and the share of the total loss that comes from the most expensive tokens. The text below is coloured by each token’s loss.
- State the pretraining objective precisely.
- Explain how one pass yields a loss at every position.
- Read the distribution of per-token losses.
- See which tokens dominate the average.
An average over very different tokens
On this English passage micro’s mean loss is 3.914 nats per token but the median is 3.510: the distribution is skewed, with a long tail of expensive tokens. 5% of the tokens cost less than half a nat — continuations the model is nearly sure of — while the costliest tenth carries 23% of the total loss. On the Chinese passage the mean is 4.365; on a plain list of numbers, where every next token follows a rule, it drops to 2.616.
Two consequences follow for everything in Part Two. First, progress in the loss comes from the hard tail — from tokens that need knowledge, long-range context or reasoning — so a small change in the average can mean a large change in what a model can do (plates VIII.5–VIII.6). Second, the objective weights every token equally: boilerplate, duplicated pages and machine-generated junk count as much as a textbook. That is why the data a model reads (Vol. IX) matters as much as how it is trained, and why the last stage of pretraining often switches to better data (Vol. XII).
Terms
- Pretraining objective
- The average next-token cross-entropy over every position of the training corpus.
- Per-token loss
- The surprisal of one token given its prefix.
- Maximum likelihood
- Choosing parameters that make the training data as probable as possible; equivalent to minimising cross-entropy.
- Skewed distribution
- Most values small, a long tail of large ones; the mean exceeds the median.