Vol. I · What a Language Model Is Plate 9 of 10pretraining, mid-training, post-training, evaluation, inference, life cycle, course mapDCLE

The Life of a Model.

Pretraining makes a predictor, mid-training sharpens it, post-training makes it useful; evaluation and serving decide what anyone gets from it.

Every large language model in use today was made in the same broad stages. Pretraining trains a randomly initialised network on next-token prediction over a very large corpus — trillions of tokens of web text, books, code and papers — and takes most of the compute. Mid-training, a name labs began to use around 2024, continues the run on smaller, chosen data: higher quality text, mathematics, code, long documents. Post-training turns the resulting base model, which only continues text, into an assistant: supervised fine-tuning on demonstrations, learning from preferences, reinforcement learning on tasks with checkable answers, and safety training.

Around these sit the stages that decide whether the model is any good and whether anyone can use it: evaluation, which measures capabilities and risks, and inference, the serving system that turns a trained network into answers at a cost per token. Underneath all of it are the data, the hardware and the parallel software that make training at this scale possible. This course follows that order.

Below: the stages of a model’s life and the course volumes that cover each. Choose a stage to see what goes in, what it optimises and roughly what share of a frontier training budget it uses. The figures for the course’s own model come from its recorded run.

What you will learn
Workshop
The life cycle of a language model, left to right, with the stage chosen in the menu highlighted. The table lists each stage’s data, objective and volumes in this course.

One map, thirty-one volumes

The course’s model micro went through the first stage only: 262 million tokens of the course corpus through 1.85 million parameters, about 142 tokens per parameter, for roughly 2.9 × 10¹⁵ floating-point operations — the 6 × parameters × tokens rule that Vol. V derives. A frontier model goes through all of them, with pretraining a hundred million times larger. Pretraining is where most of the compute goes; post-training is where most of the behaviour a user sees is shaped; mid-training sits between, and evaluation runs throughout, from the first loss curve to the safety tests before release.

The course follows the model through its life. Part One builds the network; Part Two pretrains it and studies the data and scaling laws behind that; Part Three is the hardware, parallelism, cluster operations and kernels that make a run possible; Part Four post-trains it; Part Five evaluates and serves it; Part Six turns it into an agent and builds the environments agents learn in; Part Seven looks inside and at the frontier; Part Eight is history and projects. The same small course models appear in every part, so you can watch what each stage does to them.

The word pretraining comes from the time when a network was first trained on a generic task and then fine-tuned on the one that mattered; for language models the order flipped in importance — the “pre” stage became the main event, and much of what used to be the task is now done by prompting.

Terms

Pretraining
Training on next-token prediction over a very large general corpus; most of the compute.
Base model
The model after pretraining: it continues text but does not follow instructions or converse.
Post-training
Fine-tuning, preference learning, reinforcement learning and safety training that turn a base model into an assistant.
Inference
Running a trained model to produce outputs; serving is doing it for many users at once.
A model is made in stages — pretraining, mid-training, post-training — then evaluated and served. Pretraining buys the knowledge; post-training shapes the behaviour.