Vol. X · Scaling Laws Plate 5 of 9overtraining, inference cost, total cost, small models, Llama 3 8B, Beyond ChinchillaCE

Training Past Optimal.

Spend more on training once so that every one of a trillion answers costs less.

Chinchilla’s question was: for a fixed training budget, which model is best? A deployed model is trained once and then generates tokens for months, and each generated token costs about 2N FLOPs (plate V.6). If the model will produce many tokens, a smaller one that reaches the same quality by reading more data can be cheaper overall, even though its training cost more than the compute-optimal model’s.

Sardana and colleagues (2024) made this precise: fix a target loss and an expected number of inference tokens, then choose N and D to minimise training plus inference FLOPs. As expected inference grows, the best model shrinks and its tokens per parameter rise into the hundreds or thousands. That is the logic behind small open models trained on 15 trillion tokens. The workshop does the optimisation with the refitted Chinchilla law.

Below: left, total cost (training plus inference) against model size for a fixed target quality, at the chosen lifetime inference. Right, how the best model’s size and tokens per parameter change with lifetime inference.

What you will learn
Workshop
Refitted Chinchilla law (Besiroglu et al., 2024). Target loss: that of the compute-optimal model at 10²³ FLOPs. Left: training FLOPs 6ND, inference FLOPs 2NI and their sum against N, each N paired with the D that reaches the target. Right: cost-optimal N (blue) and tokens per parameter (red) against lifetime inference I.

Pay once, save on every token

The target is the loss of the compute-optimal model at 10²³ FLOPs, 2.034, which has 29.5 billion parameters. If it will only ever generate 10¹⁰ tokens, inference is negligible and the best model is close to that one (28.8 billion parameters, 20 tokens each). At 10¹³ tokens of lifetime inference the cheapest model reaching the same loss has 11.2 billion parameters trained on 222 tokens each, saving 43% of total compute; at 10¹⁴ tokens it shrinks to 7.38 billion at 1,363 tokens per parameter.

The cost curve is again flat near its minimum, and the law’s accuracy far from the compute-optimal point is uncertain, so real choices are made with margin — and with other costs in view: a smaller model also needs less memory and responds faster, and the data to train it may not exist. Llama 3 8B, at about 1,868 tokens per parameter, sits far into this regime. The ceiling is data: past a few epochs, repeating text buys little (plate X.6).

Inference volumes are large. A service answering ten million requests a day at a thousand tokens each generates ten billion tokens a day, three and a half trillion a year — enough to make inference, not training, the dominant cost.

Terms

Overtraining
Training a model on more tokens than is compute-optimal, to make a smaller model reach a target quality.
Lifetime inference
The total number of tokens a model will generate or process in deployment.
Total cost
Training FLOPs plus inference FLOPs.
Cost-optimal model
The size that reaches a target quality at the least total cost.
A model that will serve many tokens should be smaller and trained longer than compute-optimal: training is paid once, inference on every token.

Sources

  1. Sardana, Portes, Doubov and Frankle, “Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws”, ICML 2024.
  2. Besiroglu et al., “Chinchilla Scaling: A replication attempt”, arXiv:2404.10102, 2024.