Vol. VIII · The Pretraining Run Plate 3 of 10loss curve, training loss, validation loss, epochs, overfitting, learning-rate schedule, tokens seenLD

Reading a Loss Curve.

Two lines, one on what the model studies and one on what it never sees — and the space between them.

Every training run logs two numbers over time. The training loss is the average loss on recent batches, computed anyway for the gradient. The held-out (validation) loss is measured now and then on text kept out of training. Early on they fall together and fast; then progress slows to a steady crawl, roughly a straight line on a log scale of tokens; and at the end of a cosine schedule (plate XI.1) the falling learning rate gives one last drop.

Large runs read each document once or a few times, so the two curves stay together and the held-out loss is an honest measure of learning. Small runs on a small corpus are different: the course models read the same 15 million training tokens many times over, and the curves separate as the model starts to fit the particular text it has seen. The workshop plots the recorded curves of micro and of mini, a model eight times larger trained on the same corpus.

Below: training loss (light) and held-out loss (dark) of the selected recorded run against tokens seen, with a mark at every pass through the training data; and the learning rate on the same axis.

What you will learn

Distinguish training loss from held-out loss.
Read the phases of a loss curve.
Count epochs from tokens seen and tokens available.
Recognise the gap that repeated data opens.

Workshop

Recorded runs on the course corpus (15.2 million training tokens). Left: training loss averaged over each logging interval and held-out loss at each evaluation, against tokens seen; vertical lines mark each completed epoch. Right: the learning-rate schedule.

Two lines and the space between

micro reads its 15.2 million training tokens 17.2 times over its 32 000 steps. Its held-out loss falls from 4.69 at the first evaluation to 3.948 at the end, with the steepest progress in the first few million tokens and a final drop as the cosine schedule brings the learning rate down. The training loss ends at 3.791, below the held-out loss by 0.156 nats: after so many passes the model has started fitting the particular sentences it has read, which helps nothing on new text.

mini has eight times the non-embedding parameters and reads the data 6.5 times; it ends at 3.712 held-out, against micro’s 3.948, but with a wider gap of 0.366 nats against micro’s 0.156 — a larger model memorises repeated text faster (plate VIII.7). Real pretraining runs avoid this regime: Llama 3 read about 15 trillion tokens, mostly once, so its training and held-out curves are nearly the same line. When text does run short, repeating it a few times costs little; repeating it dozens of times, as here, does (plate X.6).

A training loss averaged over a logging interval lags the model: it includes batches from before the latest updates. Held-out loss at a single checkpoint does not, which is one reason the two lines can cross early in training.

Terms

Training loss: The average loss on the batches the model is trained on.
Held-out loss: The loss on text never used for training; also called validation loss.
Epoch: One complete pass through the training data.
Generalisation gap: Held-out loss minus training loss; grows as the model memorises its training text.

A loss curve has two lines: when they part, the model has started learning its training text rather than the language.