Vol. X · Scaling Laws Plate 3 of 9Chinchilla, compute-optimal, tokens per parameter, IsoFLOP, Gopher, budget allocationLC

Compute-Optimal Training.

With a fixed budget, a bigger model reads fewer tokens — and somewhere between too big and too small sits the best loss.

Training compute is about 6 N D floating-point operations (plate V.6). With the budget C fixed, every extra parameter means fewer tokens, and the parametric law of plate X.2 has a minimum along that trade-off: the compute-optimal model size N*(C) and token count D*(C). In 2022 Hoffmann and colleagues found that both should grow roughly in proportion to √C, and trained Chinchilla — 70 billion parameters on 1.4 trillion tokens — with the same compute as their 280-billion-parameter Gopher, which had read 300 billion. Chinchilla was better on nearly every evaluation.

The lesson was taken as “about 20 tokens per parameter”, Chinchilla’s own ratio, which their first two estimation methods supported. Their third, the parametric fit, implies something different when its published constants are used directly; Besiroglu and colleagues traced the discrepancy to the fit and their corrected constants agree with the 20. The workshop computes the optimal split for any budget under both sets of constants.

Below: left, predicted loss against model size at the chosen budget (an IsoFLOP curve) under both sets of constants, with their minima. Right, tokens per parameter of published models.

What you will learn
Workshop
Left: loss predicted by L = E + A/N^α + B/D^β along 6ND = C for the chosen C, with Hoffmann et al.’s published constants (blue) and Besiroglu et al.’s refit (red); dots mark the minima. Right: tokens per parameter for published models: Gopher, Chinchilla, GPT-3, Llama 3 8B and Llama 3 405B.

Too big, too small, and the 20

At Chinchilla’s budget of about 5.8 × 10²³ FLOPs, the refitted constants put the optimum at 72.3 billion parameters with 18 tokens each — close to Chinchilla’s actual 70 billion and 20. Under the same constants, Gopher’s 280 billion parameters on 300 billion tokens predict a loss of 2.001 and Chinchilla’s split 1.976: same compute, better model. Hoffmann’s published constants, used directly, put the optimum at 32.2 billion with 93 tokens per parameter, which disagrees with their own other methods — the inconsistency Besiroglu and colleagues resolved.

The IsoFLOP curve is flat near its minimum: a model somewhat larger or smaller than optimal loses little. That flatness is what makes deliberate departures cheap, and nearly every modern model departs on purpose. Llama 3 8B read about 1,868 tokens per parameter, far past the optimum, because a smaller model is cheaper to serve (plate X.5). At Llama 3 405B’s budget of 3.8 × 10²⁵ FLOPs the refit gives 619 billion parameters at 17 tokens each; Meta’s own scaling law suggested 402 billion on 16.55 trillion tokens, and they trained 405 billion on 15.6 trillion.

The optimal ratio is not a constant of nature: it depends on the data, the architecture and the tokeniser, and Meta’s fit for Llama 3 implied about 40 tokens per parameter at its budget. “Twenty” is a summary of one study, not a law.

Terms

Compute-optimal
The split of a fixed budget between N and D that minimises predicted loss.
IsoFLOP curve
Loss against model size at a fixed compute budget.
Tokens per parameter
D / N; about 20 for Chinchilla.
Chinchilla
DeepMind’s 70-billion-parameter model of 2022, trained on 1.4 trillion tokens.
For a fixed budget, loss has a minimum between too big and too small — near 20 tokens per parameter for Chinchilla, flat enough that modern models depart from it on purpose.

Sources

  1. Hoffmann et al., “Training Compute-Optimal Large Language Models”, NeurIPS 2022.
  2. Besiroglu, Erdil, Barnett and You, “Chinchilla Scaling: A replication attempt”, arXiv:2404.10102, 2024.
  3. Llama Team, AI @ Meta, “The Llama 3 Herd of Models”, arXiv:2407.21783, 2024.