Vol. X · Scaling Laws Plate 8 of 9irreducible loss, diminishing returns, data wall, Villalobos, regime change, limits of scalingCE

Where Scaling Laws Break.

Each tenfold of compute buys less than the last, the floor never moves, and somewhere ahead the text runs out.

A power law with a floor has diminishing returns built in: each tenfold increase in compute removes a fixed fraction of the reducible loss, never the floor itself. Laws also hold only while the recipe holds. Changes of architecture, data or objective move whole curves — better data lowers the line rather than sliding along it — and at some scales new behaviour has appeared that the curve did not predict.

The most concrete limit is data. Villalobos and colleagues estimated the stock of usable public human text at about 300 trillion tokens (90% interval 100 to 1 000 trillion) and projected that the largest training runs would use it all between 2026 and 2032, earlier if models are overtrained. The workshop follows the compute-optimal frontier of the refitted Chinchilla law and marks where its data requirement meets that stock.

Below: left, loss along the compute-optimal frontier as compute grows, approaching the floor E. Right, the tokens a compute-optimal run needs at each budget, against the estimated stock of human text.

What you will learn
Workshop
Refitted Chinchilla law (Besiroglu et al., 2024). Left: compute-optimal loss against compute, and the floor E (dashed). Right: compute-optimal tokens D* against compute, with the estimated stock of usable public human text (Villalobos et al., 2024): 300 trillion tokens, shaded from 100 to 1 000 trillion.

Diminishing returns and a finite supply

Along the compute-optimal frontier of the refitted law, the loss above the floor shrinks as a power of compute with exponent 0.178: at 10²⁴ FLOPs a further tenfold buys 0.048 nats, at 10²⁸ only 0.009. At 10²⁶ the law predicts 1.882, still 0.062 above its floor of 1.82. Diminishing returns in loss do not by themselves mean diminishing returns in usefulness — small losses can carry large changes in skill (plate X.7) — but every further step costs ten times more.

The data line meets the stock of human text — about 300 trillion tokens — at about 10^28.8 FLOPs for compute-optimal training, between 10^27.8 and 10^29.8 across the stock’s uncertainty; overtrained models reach it sooner. That is why labs have turned to repeated data within its limits (plate X.6), synthetic and verified data (plate IX.9), other modalities (Vol. XXIX), and spending compute in new places — reinforcement learning and inference-time reasoning (Vols. XIX and XXX) — that are not bound by the pretraining curve. Whether those keep progress on a similar trend is one of the open questions of Vol. XXX.

Laws also break from below: a change of recipe, such as better data filtering, shifts the curve down, so a fitted law describes one recipe at one time. Comparing models trained years apart against a single law mixes the two kinds of progress.

Terms

Diminishing returns
Each multiple of a resource buying a smaller absolute gain.
Floor
The irreducible loss E that a law approaches but never passes.
Data wall
The scale at which a run needs more usable text than exists.
Regime change
A shift in recipe or behaviour that moves or bends the fitted curve.
Scaling laws have a floor, a recipe and a fuel supply: returns diminish, the curve moves when the recipe changes, and human text is finite.

Sources

  1. Villalobos et al., “Will we run out of data? Limits of LLM scaling based on human-generated data”, ICML 2024, arXiv:2211.04325.
  2. Besiroglu et al., “Chinchilla Scaling: A replication attempt”, arXiv:2404.10102, 2024.