Vol. XXXI · History and Projects Plate 4 of 14history, scaling, GPT-3, scaling laws, Chinchilla, PaLM, Llama, DeepSeek-V3, tokens per parameterDCE

The Age of Scale.

First the models grew. Then the data caught up.

GPT had 117 million parameters in 2018; GPT-3 had 175 billion two years later, a 1,496-fold increase, trained on 300 billion tokens — under 1.7 tokens per parameter. Kaplan and colleagues (2020) had found that loss falls as a power law in parameters, data and compute, and their fits favoured spending extra compute mostly on parameters.

Hoffmann and colleagues (2022) redid the fits with learning-rate schedules matched to each run’s length and found parameters and tokens should grow together, about 20 tokens per parameter: their 70-billion-parameter Chinchilla, trained on 1.4 trillion tokens, beat larger models. Since then the ratio has gone further, because a smaller model trained longer is cheaper to serve (plate X.5). The workshop plots the models by year and size, and their tokens per parameter.

Left: parameters of landmark models by year, on a logarithmic scale. Right: training tokens per parameter for the models whose token counts were published.

What you will learn
Workshop
Published parameter counts and training-token counts. DeepSeek-V3 is a mixture of experts: 671 B total, 37 B active per token; its ratio is shown per active parameter when selected. PaLM: 780 B tokens; Llama 3 405B: 15.6 T tokens; DeepSeek-V3: 14.8 T tokens.

From bigger models to longer training

GPT-3 saw 1.7 tokens per parameter and PaLM 1.4: by Chinchilla’s fits both were far too large for their data. Chinchilla itself used 20.0. Llama 3 405B used 38.5, about twice the compute-optimal ratio, and its smaller siblings went much further, because inference cost depends on parameters and is paid on every token served, while training is paid once (plate X.5).

The headline growth in parameters stalled at a few hundred billion dense parameters; growth continued instead in tokens, in compute per token through mixture-of-experts — DeepSeek-V3 activates 37 billion of its 671 billion parameters per token — and, from 2024, in compute spent at inference and in reinforcement learning (Vol. XXX). The scaling laws of Vol. X still hold in the form they were measured; what changed is which quantity it pays to scale.

Kaplan’s and Hoffmann’s different conclusions came largely from one choice: Kaplan used a fixed learning-rate schedule length, so shorter runs were measured before they had finished decaying.

Terms

Scaling law
A power-law fit of loss to parameters, data or compute.
Compute-optimal
The split of a compute budget minimising loss.
Tokens per parameter
Training tokens divided by parameters.
Over-training
Training beyond the compute-optimal ratio to cut inference cost.
Scaling first meant more parameters, then more tokens per parameter, as Chinchilla’s fits and inference costs changed what it paid to grow.

Sources

  1. Kaplan et al., “Scaling Laws for Neural Language Models”, arXiv:2001.08361, 2020.
  2. Brown et al., “Language Models are Few-Shot Learners”, NeurIPS 2020.
  3. Hoffmann et al., “Training Compute-Optimal Large Language Models”, NeurIPS 2022.
  4. Chowdhery et al., “PaLM: Scaling Language Modeling with Pathways”, 2022.
  5. Llama Team, “The Llama 3 Herd of Models”, arXiv:2407.21783, 2024.
  6. DeepSeek-AI, “DeepSeek-V3 Technical Report”, arXiv:2412.19437, 2024.