Vol. X · Scaling Laws Plate 1 of 9power law, log–log plot, scaling exponent, Kaplan, fitting, extrapolationLC

Power Laws.

Double the data, and the loss falls by the same factor every time — a straight line, once both axes are logarithmic.

In 2020 Kaplan and colleagues trained hundreds of language models and found that held-out loss fell as a power of each resource when the others were not limiting: L ∝ N^−0.076 in parameters, L ∝ D^−0.095 in tokens and L ∝ C^−0.050 in compute, over many orders of magnitude. A power law is a straight line on log–log axes, and its slope is the exponent: doubling D multiplies the loss by 2^−0.095, about 0.94, whatever D is.

Straight lines can be fitted on cheap runs and extended, which is why this regularity reshaped how models are planned. The course’s recorded grid trains four model shapes, from 25 thousand to 444 thousand non-embedding parameters, on 1 to 14 million tokens each, every run within a single pass over the corpus. The workshop fits a power law in tokens for each shape.

Below: held-out loss against tokens on log–log axes for the four recorded shapes, with the fitted line for the selected one. Its slope is the data exponent.

What you will learn

Recognise a power law as a straight line on log–log axes.
Fit an exponent from a few runs.
Interpret an exponent as a multiplicative gain per doubling.
Compare fitted exponents with published ones.

Workshop

Recorded runs, each 1 to 14 million tokens at batch 32 × 128 with a cosine schedule ending at the last step; loss on held-out course text. Both axes logarithmic. The dashed line is the least-squares fit of log L on log D for the selected shape.

Straight lines on logarithmic axes

For the 3-layer, 64-wide model the five runs fall close to a line on log–log axes, with R² = 0.991. Its slope gives a data exponent of 0.124: each doubling of tokens multiplies the loss by 0.918. The smallest shape has an exponent of 0.102 and the largest 0.123 — a bigger model gains more from each extra token, because it has the capacity to use it. Kaplan and colleagues’ exponent for data was 0.095, a gain of 0.936 per doubling; the course’s exponents are steeper because these runs are tiny and short, far from the regime where their law was measured, and the line would bend as the model saturated.

That bend is the main caution of this volume. A power law fitted over a range says nothing certain outside it: extending this line to 100 million tokens predicts 3.75 nats, which the model’s limited capacity would not reach. Later work therefore adds an irreducible term — a floor that no amount of data removes — and fits loss in parameters and data jointly (plate X.2). The regularity is still remarkable: across sizes and budgets, a few numbers summarise how a whole family of models responds to more data.

Why power laws? One explanation treats language as a long tail of patterns, each learned once enough examples have been seen; if pattern frequencies follow a power law, as word frequencies do, the loss left over follows one too. It is a hypothesis, not a theorem.

Terms

Power law: y = a x^−b; a straight line of slope −b on log–log axes.
Scaling exponent: The power b: how fast loss falls as a resource grows.
Log–log plot: Both axes logarithmic, so equal ratios are equal distances.
Extrapolation: Using a fitted law outside the range it was fitted on.

Loss falls as a power of each resource — a straight line on log–log axes, whose slope is fitted on small runs and trusted only with care beyond them.

Sources

Kaplan et al., “Scaling Laws for Neural Language Models”, arXiv:2001.08361, 2020.