Vol. XII · Mid-Training Plate 1 of 10mid-training, training stages, annealing, long-context extension, data curriculum, Llama 3, OLMo 2DL

What Mid-Training Is.

A run is no longer one stream of data: after the long stretch on the web come shorter stages with different data, longer documents and a falling learning rate.

Early large models were pretrained on one data mix from start to finish. Recent ones are trained in stages. Llama 3 405B, by its report, read about 15.6 trillion tokens in all: most of them in its main mix; about 800 billion in six stages that stretched its context from 8 thousand to 128 thousand tokens; and finally 40 million of carefully chosen data while the learning rate was annealed to zero and checkpoints were averaged. OLMo 2 split its pretraining into a web-dominated first stage and a mid-training stage on a smaller, higher-quality mix.

The name covers everything between the main pretraining run and post-training: changing the data mix towards quality, code and mathematics; extending the context; adapting to new domains or languages; and choosing how the learning rate ends. The stages are small in tokens but large in effect, because they come last, when the learning rate is falling and the model consolidates what it is shown (plate XI.1). The workshop puts Llama 3’s published stages on one axis.

Below: left, the tokens in each stage of Llama 3 405B’s pretraining, on a log scale. Right, each stage’s share of the total — the final stages are a sliver of the tokens.

What you will learn

Name the stages of a modern pretraining run.
Compare their sizes in tokens.
Explain why late stages have an outsized effect.
Place mid-training between pretraining and post-training.

Workshop

Llama 3 405B’s pretraining stages as reported (arXiv:2407.21783, sections 3.4.1–3.4.3), of about 15.6 trillion tokens in all: long-context stages about 800 billion; final annealing 40 million; main pretraining the remainder. Left: tokens per stage, log scale. Right: shares of the total.

Small stages at the end

In Llama 3 405B’s reported recipe the long-context stages are 5.1% of the pretraining tokens and the final annealing 0.0003% — the main run is about 370 thousand times larger than the annealing stage. Yet each late stage changes the model visibly: the 6 long-context stages take the usable context from 8,192 to 131,072 tokens (plate XII.4), and annealing on high-quality data raised the 8B model’s grade-school mathematics score by about a quarter (plate XII.2).

Late stages matter more than their size because of where they sit. As the learning rate falls the model stops moving far and settles into whatever its latest data favour, so the data seen during the decay shapes the final weights disproportionately. Labs exploit this deliberately: the expensive, scarce data — curated textbooks, verified mathematics, long documents — is saved for the end, and several annealing runs can branch from one checkpoint to compare mixes cheaply. Mid-training also blurs into post-training, since instruction-like data increasingly appears in its mix (Vol. XVII).

Releasing a “base model” therefore usually means releasing a model after mid-training, not after the web-only phase; the base model of plate VIII.9 may already have seen curated mathematics, code and some instruction-like text.

Terms

Mid-training: The stages between the main pretraining run and post-training: data shifts, context extension, annealing.
Annealing: Lowering the learning rate to zero at the end of pretraining, usually on higher-quality data.
Stage: A part of training with its own data mix, context length or schedule.
Curriculum: An ordering of training data, here from broad to curated.

Modern pretraining ends in small, late stages — better data, longer contexts, a falling learning rate — whose effect far exceeds their share of tokens.

Sources

Llama Team, AI @ Meta, “The Llama 3 Herd of Models”, arXiv:2407.21783, 2024, sections 3.1.3 and 3.4.
Team OLMo, “2 OLMo 2 Furious”, arXiv:2501.00656, 2024.