Vol. X · Scaling Laws Plate 6 of 9data-constrained scaling, epochs, repetition, overfitting, Muennighoff, weight decayLD

When Data Runs Out.

Read the same text four times and it is nearly as good as new; read it a hundred times and the model learns the text instead of the language.

Every law in this volume so far assumes that each token is new. Real budgets can outrun the supply of good text (plate X.8), and then data must be repeated. Muennighoff and colleagues (2023) trained hundreds of models up to 9 billion parameters with repeated data and found that, at fixed compute, up to about four epochs of repetition gave almost the same loss as unique data; beyond that, each further repetition was worth less, and eventually nothing.

The course has a sharp version of the experiment. Three recorded runs of micro’s shape spend identical compute — 6 000 steps, 49 million tokens — on different amounts of unique text: the full 15.2 million training tokens (3.2 epochs), 3 million (16 epochs), and 300 thousand (164 epochs). A fourth repeats the smallest with weight decay ten times stronger.

Below: held-out loss over training for the four runs. Right, the best and final held-out loss of each, and its number of epochs.

What you will learn

Count epochs from tokens seen and unique tokens.
Compare runs of equal compute with different unique data.
Recognise overfitting from repeated data.
State the published limit on useful repetition.

Workshop

Recorded runs: micro’s shape, 6 000 steps of 64 × 128 tokens (49 million tokens seen), cosine schedule. Unique training tokens: 15.2 million, 3 million, 300 thousand, and 300 thousand with weight decay 1.0. Held-out loss on text none of them trained on.

Repetition has a price

With the full 15.2 million unique tokens, read 3.2 times, micro’s shape ends at 4.208 nats. With 3 million, read 16 times, it ends at 5.215 — a clear loss for the same compute. With 300 thousand tokens read 164 times the run collapses: held-out loss bottoms out at 7.35 early in training and then climbs to 16.28, far worse than predicting token frequencies alone, because the model has memorised its tiny corpus and assigns near-zero probability to anything else. Ten times the weight decay slows the damage — it ends at 10.39 — but does not prevent it.

These are far more epochs than any large run would use, which is why the effect is so stark; Muennighoff and colleagues’ result marks the safe end: up to about 4 epochs, repetition is nearly free. micro itself, at 17 epochs and 262 million tokens seen, still improved to 3.948 because it had more compute to spend, but its widening gap between training and held-out loss (plate VIII.3) shows the same pressure. When text runs short, the responses are more epochs within that limit, more careful filtering to keep the best text for repetition, other domains such as code, and synthetic data (plate IX.9).

The held-out loss of the 300-thousand-token run rises above the unigram baseline of about 7.6 nats: a model that has memorised a small text is confidently wrong about new text, which is worse than knowing nothing.

Terms

Epoch: One pass through the unique training data.
Data-constrained: A budget that would need more unique tokens than exist.
Repetition: Reading the same tokens more than once during training.
Overfitting: Fitting the training text so closely that held-out loss rises.

Repetition is nearly free up to about four epochs and increasingly costly beyond; with far too little data a model memorises it and gets worse on everything else.

Sources

Muennighoff et al., “Scaling Data-Constrained Language Models”, NeurIPS 2023.