Drills: Mid-Training.
Five questions at a time from the whole volume, until the end of pretraining is as familiar as its beginning.
This plate draws rounds of five from the questions and predictions of the nine plates before it, with the reasoning and source plate after each answer.
The volume’s thread is leverage at the end of a run: data, context and averaging applied while the learning rate falls shape the final model far more than their share of tokens, and every specialisation trades against what was learned before.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Name the stages of mid-training.
- Reason about annealing and upsampling.
- Explain forgetting and replay.
- Explain distillation, averaging and the cutoff.
What to carry forward
Pretraining now ends in small, late stages (plate XII.1): annealing on high-quality data decides where the model settles (plate XII.2), scarce domains are upsampled where they count most (plate XII.3), and context is extended in stages with rescaled positions (plate XII.4).
Continued pretraining moves a model to a new domain at some cost to the old (plate XII.5), replay prevents most forgetting (plate XII.6), distillation passes a teacher’s full distribution to a student (plate XII.7), averaging cancels noise (plate XII.8), and the knowledge cutoff ages from release onwards (plate XII.9). Part Three turns to the machines that make all of this possible.
Terms
- Annealing
- Decaying the learning rate at the end, on chosen data.
- Replay
- Mixing old data into a new phase.
- Distillation
- Matching a teacher’s distribution.
- Knowledge cutoff
- The date of the newest training text.