Drills: Scaling.
Five questions at a time from the whole volume, until 6ND, the 20 and the floor are second nature.
This plate draws rounds of five from the questions and predictions of the eight plates before it, with the reasoning and source plate after each answer.
The volume’s thread is planning under a budget: how loss responds to parameters, data and compute, how to split a budget, when to depart from the split on purpose, and what the curves cannot promise.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Read and fit power laws.
- Compute compute-optimal splits and their caveats.
- Reason about overtraining and repetition.
- State the limits of scaling laws.
What to carry forward
Loss falls as a power of each resource — straight lines on log–log axes (plate X.1) — and jointly as a floor plus a size penalty plus a data penalty whose constants are hard to pin down (plate X.2). For a fixed budget C ≈ 6ND the loss has a flat minimum, near 20 tokens per parameter for Chinchilla (plate X.3), and the same laws fitted on small runs forecast large ones with errors that grow with distance (plate X.4).
Deployed models are deliberately overtrained to cut inference cost (plate X.5), up to the limits of repeated data — about four epochs nearly free (plate X.6). Many skills track loss across sizes, which lets benchmarks be forecast in two steps (plate X.7), and the laws themselves meet a floor, changing recipes and a finite stock of text (plate X.8). Vol. XI turns to the optimisers and numerics that make such large runs train at all.
Terms
- Power law
- A straight line on log–log axes.
- Compute-optimal
- The best split of a fixed budget.
- Overtraining
- More tokens than optimal, for cheaper inference.
- Data wall
- Needing more text than exists.