Drills: Optimisation.
Five questions at a time from the whole volume, until the knobs of a large run are familiar.
This plate draws rounds of five from the questions and predictions of the nine plates before it, with the reasoning and source plate after each answer.
The volume’s thread is stability and efficiency: settings that let a run take large, cheap steps without diverging, and the measurements that show when it is about to.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Choose and explain a learning-rate schedule.
- Reason about batch size and Adam’s settings.
- Read gradient norms and loss spikes.
- Explain the number formats of training.
What to carry forward
Decaying the learning rate lowers the noise floor, and WSD postpones the decay so runs can be extended (plate XI.1). Larger batches need fewer steps until the critical batch (plate XI.2). Adam’s β₂ and ε shape its response to sudden and tiny gradients (plate XI.3); clipping caps steps but not causes (plate XI.4); and spikes are handled by rollback and prevented by removing the instability (plate XI.5).
Training runs in BF16 with FP32 master weights, and FP8 with fine-grained scales (plates XI.6–XI.7). μP makes small proxies faithful so that hyperparameters transfer (plate XI.8), and matrix-aware optimisers such as Shampoo and Muon have reopened the question of Adam’s place (plate XI.9). Vol. XII turns to what happens near the end of pretraining: the data and stages of mid-training.
Terms
- WSD
- Warmup–stable–decay schedule.
- Critical batch
- Where bigger batches stop paying.
- BF16
- 8 exponent bits, 7 mantissa bits.
- μP
- Width-aware parametrisation.