Vol. X · Scaling Laws Plate 4 of 9extrapolation, prediction, small runs, scaling law fitting, GPT-4, Llama 3, forecast errorLC

Predicting From Small Runs.

Fit on the cheap runs, predict the expensive one — and check how far off the forecast lands.

Before a large run, labs train families of small models with the same recipe and fit a law to them. GPT-4’s technical report (2023) says its final loss was predicted from models trained with at most one ten-thousandth of its compute; Llama 3’s team fitted runs from 6 × 10¹⁸ to 10²² FLOPs and extrapolated to 3.8 × 10²⁵, more than three thousand times the largest fitted run, to choose the flagship’s size.

Extrapolation errors grow with distance, and they grow faster when the fit’s constants are poorly pinned down (plate X.2). The workshop does the exercise on the course grid: fit the law on the smallest shapes only, predict the largest shape’s losses, and compare with what it actually recorded.

Below: left, the largest shape’s recorded losses against the predictions of a law fitted on the smaller shapes only. Right, the prediction errors.

What you will learn
Workshop
The law L = E + A/N^α + B/D^β fitted in the page to the recorded runs of the smallest shapes only (the chosen number of shapes, all token budgets), then used to predict the 4-layer, 96-wide model (444 thousand non-embedding parameters) at 1, 2, 4 and 8 million tokens.

Forecasts and their error bars

Fitted on the three smaller shapes, the law predicts the largest — 3.0 times more non-embedding parameters than the largest shape it saw — with a mean error of 0.071 nats and at worst 0.160, against losses between 5 and 6.5. Fitted on only the two smallest, the extrapolation stretches to 8.0 times and the mean error becomes 0.093. A wider fitting range and a shorter extrapolation give a better forecast; how much better depends on how well the fit’s constants are determined, which here is not well (plate X.2).

Large labs work at a different scale but with the same logic. Llama 3’s team extrapolated about 3,800 times beyond its largest fitted run in compute, and complemented the loss law with a second fit relating loss to benchmark accuracy (plate X.7). The useful habit is to treat a forecast as a range: fit on several subsets, see how far the predictions scatter, and plan for the spread rather than for the single number.

Predicting loss is easier than predicting abilities. GPT-4’s report also predicted pass rates on a coding benchmark from small runs, but noted that some capabilities remained hard to predict — a theme of plates VIII.6 and X.7.

Terms

Extrapolation distance
How far beyond the fitted runs a prediction reaches, in parameters or compute.
Forecast error
The difference between a prediction and the measured result.
Fitting range
The runs a law is fitted on.
Recipe
The fixed choices — data, architecture, optimiser, schedule — shared by every run of a scaling study.
Scaling laws are used to predict an expensive run from cheap ones; the error grows with the distance extrapolated and with uncertainty in the fit.

Sources

  1. OpenAI, “GPT-4 Technical Report”, arXiv:2303.08774, 2023, section 3.1 (final loss predicted from runs using at most 1/10 000 of the compute).
  2. Llama Team, AI @ Meta, “The Llama 3 Herd of Models”, arXiv:2407.21783, 2024, section 3.2.1.