Vol. X · Scaling Laws Plate 7 of 9downstream performance, loss to accuracy, sigmoid, probes, benchmark prediction, emergence and lossLE

From Loss to Skills.

Plot a skill against loss rather than against size, and models of every size fall on one curve.

A lower loss is only useful if it shows up as better behaviour. A useful empirical regularity is that, for many skills, the score is close to a function of pre-training loss alone: models of different sizes, trained on different amounts of data, that reach the same loss tend to score about the same. Llama 3’s team used this in two steps — a scaling law from compute to loss on a benchmark’s answers, then a fitted sigmoid from that loss to accuracy — to forecast the flagship’s benchmark scores. Du and colleagues (2024) argued that apparent emergence is often a skill appearing once loss falls below a threshold.

The course has the data to check it. Every probed checkpoint — the scaling grid, the mixture runs, micro’s shape and the eight-times-larger mini — recorded its English held-out loss alongside the agreement probe (choosing “is” or “are” to match a subject) and the induction gain (plate VIII.5). The workshop plots each skill against loss, not against model size.

Below: left, the agreement probe’s score against English held-out loss for every probed checkpoint, coloured by model family. Right, the induction gain against the same loss.

What you will learn
Workshop
Every recorded checkpoint with probes: 19 runs of the scaling grid (final), four mixture runs, micro’s shape and mini (40 checkpoints each). x: held-out English loss; y: agreement (left; chance is 0.5) and induction gain in nats (right). The dashed line marks the chosen loss threshold.

One curve for every size

Across 99 checkpoints from models of 25 thousand to 8.4 million parameters, agreement sits at chance while English loss is above about 5.8 nats (mean 0.50), and is essentially perfect below 4.2 (mean 1.00), whichever model produced the checkpoint. A small model that reached loss 5 by reading more and a larger one that reached it early in training score about the same: for this skill, loss is the variable that matters, not size or data separately. That is what makes two-step forecasting work — predict loss from compute, then accuracy from loss.

Induction is different. Near a loss of 4.2, mini’s checkpoints show an induction gain of 0.32 nats and micro’s 0.23: not far apart, but mini goes on to 1.11 at lower loss while micro never does, because mini has more layers to build the copying circuit. Some skills depend on architecture as well as loss, and the loss on generic text is not a perfect proxy for a specific capability. Forecasts of benchmarks therefore fit their own loss — on the benchmark’s answers — and their own sigmoid, and are less reliable than forecasts of loss itself.

The sigmoid’s steep middle is where a skill appears to “emerge”: a modest fall in loss carries the score from chance to near the ceiling. Seen against loss rather than size, the jump is the same smooth curve as in plate VIII.6.

Terms

Downstream performance
Scores on tasks people care about, as opposed to loss.
Two-step forecast
Compute to loss with a scaling law, then loss to accuracy with a fitted curve.
Loss threshold
The loss below which a skill reliably appears.
Proxy
A quantity used in place of the one of interest; loss is a proxy for capability.
Many skills depend on loss rather than on size or data separately, so benchmarks can be forecast in two steps — though not every skill follows loss alone.

Sources

  1. Llama Team, AI @ Meta, “The Llama 3 Herd of Models”, arXiv:2407.21783, 2024, section 3.2.1 (two-step forecast of benchmark accuracy).
  2. Du, Zeng, Dong and Tang, “Understanding Emergent Abilities of Language Models from the Loss Perspective”, arXiv:2403.15796, 2024.