Vol. X · Scaling Laws Plate 2 of 9scaling grid, parametric loss, irreducible loss, Chinchilla form, fitting, parameters and dataLC

Model Size and Data.

A floor that never moves, plus a penalty for too few parameters, plus a penalty for too little data.

Neither parameters nor data alone sets the loss. A small model saturates no matter how much it reads; a large model learns little from a few tokens. Hoffmann and colleagues (2022) summarised the joint dependence in one form: L(N, D) = E + A/N^α + B/D^β. E is the irreducible loss — the entropy of text itself, which no model removes — and the two power-law terms are the penalties for finite size and finite data.

The form is simple; fitting it is not. Its five constants trade off against each other, and different fitting choices give noticeably different values. Besiroglu and colleagues (2024) refitted Hoffmann’s own data and obtained a larger E, a much larger B and a larger β, which changed what the law implied for model planning (plate X.3). The workshop shows the course’s grid of 19 runs, and fits the same form to it in the page.

Below: left, the grid of final held-out losses, model size by token budget. Right, loss against parameters for each budget, with the fitted law’s curves.

What you will learn

Read a scaling grid of size against data.
Write the parametric loss form and name its terms.
Fit the form to recorded runs.
Explain why its constants are hard to pin down.

Workshop

Left: final held-out loss of 19 recorded runs — four shapes (non-embedding parameters) by five token budgets; the largest shape was not run at 14 million tokens. Right: measured losses (points) and the form fitted to all 19 runs in the page (lines), against non-embedding parameters, one line per budget.

Two limits and a floor

Across the grid, data does most of the work. At 8 million tokens, an 18-fold increase in parameters lowers the loss by 0.41 nats; for the 148-thousand-parameter model, a 14-fold increase in tokens lowers it by 1.79. These runs are far from saturating their models, so the data term dominates. Fitting the full form to all 19 runs gives α = 0.300, β = 0.125 and a floor E of 0.00 — a floor of zero, which is impossible for real text, at the edge of what the fit allows.

Fix the floor and the fit barely worsens: at E = 2 the error is 0.073 nats and β becomes 0.200; at E = 3.5, 0.088 and 0.325. The data cannot tell these apart, and each choice predicts something different far outside the grid. The same fragility appears at scale: Besiroglu and colleagues found that Hoffmann’s published constants did not fit Hoffmann’s own runs well, and their refit implies a different model size for the same budget (plate X.3). Scaling laws are fitted, not derived, and their uncertainty should travel with every extrapolation.

The irreducible loss E is often read as the entropy of natural text under the tokeniser. It is better read as the loss this model family would reach with unlimited size and data — a property of the architecture and the data as much as of language.

Terms

Irreducible loss E: The floor of the fitted law; what unlimited parameters and data would still leave.
Parametric fit: Choosing the constants of an assumed formula to match measured runs.
Scaling grid: Runs at several sizes and several data budgets.
Identifiability: Whether the data pin down a fit’s constants uniquely; here they do not.

Loss is a floor plus a size penalty plus a data penalty — easy to write, hard to fit, and uncertain when extended.

Sources

Hoffmann et al., “Training Compute-Optimal Large Language Models”, NeurIPS 2022, section 3.3.
Besiroglu, Erdil, Barnett and You, “Chinchilla Scaling: A replication attempt”, arXiv:2404.10102, 2024.