Vol. IX · Data Plate 9 of 12synthetic data, model collapse, recursive training, distribution tails, data accumulation, textbooksLD

Synthetic Data.

Train each generation on the last one’s writing and the rare words go first — unless the real data is kept.

Synthetic text enters training in two ways. Deliberately: labs generate textbook-style explanations, exercises, rephrasings of web pages and worked solutions, filtered for quality, because they can target exactly what a model lacks; Microsoft’s phi-1 (2023) was trained largely on such “textbook quality” data. And by accident: as more of the web is written with model help, crawls collect it whether anyone wants it or not.

Shumailov and colleagues (Nature, 2024) showed what happens when models are trained recursively on their predecessors’ output: each generation samples a finite amount of text, rare events are under-sampled, and the next model never learns them. Over generations the tails of the distribution disappear and the output narrows — model collapse. Later work found that if each generation’s data is added to the real data rather than replacing it, the decline is much smaller. The workshop runs both regimes with word-bigram models on the course corpus.

Below: left, the share of the real text’s rare words (seen at most three times) that each generation still produces, when each generation trains only on the last one’s output and when output is accumulated with the real text. Right, the cross-entropy each generation assigns to real held-out text.

What you will learn
Workshop
Word-bigram models. Generation 0 is fitted to 60 000 characters of course text; each later generation is fitted to words sampled from the one before (red, replace) or to the real text plus every sample so far (blue, accumulate). Left: share of the real text’s rare words still produced by each generation’s sample. Right: cross-entropy of 12 000 characters of other course text under each generation (nats per word).

The tails go first

The real text has 1,349 rare words — words it uses at most three times. Generation 0’s sample of 20,000 words produces 87% of them. When each generation learns only from the last one’s sample, a rare word that happens not to be sampled is gone for good, and the share falls generation by generation to 37% at generation 8; the number of distinct words drops from 1,710 to 959, and the cross-entropy of real held-out text rises from 6.84 to 7.17 nats per word. With smaller samples of 5 000 words the loss is faster, 11% by generation 8.

Accumulating — keeping the real text and adding each generation’s output — holds the rare words at 81%, because they remain in the training data. It does not make the synthetic text harmless: the cross-entropy still rises, to 7.16, as synthetic text comes to outweigh the real. The practical lessons match the published ones. Keep real data in the mix; filter and verify synthetic data rather than taking it raw; and know where text came from, which is why the provenance of web text after 2022 has become a concern of its own.

Synthetic data works best where correctness can be checked: generated code can be run against tests and generated mathematics checked against answers, so only verified examples are kept. This idea returns in Vol. XIX.

Terms

Synthetic data
Training text written by a model rather than a person.
Model collapse
Loss of the distribution’s tails, and narrowing of output, when models are trained recursively on generated data.
Accumulation
Adding generated data to the real data instead of replacing it.
Provenance
Where a piece of text came from, and whether a model wrote it.
Trained on its own output, a model loses the rare parts first; keeping real data and verifying synthetic data are the defences.

Sources

  1. Shumailov et al., “AI models collapse when trained on recursively generated data”, Nature 631, 755–759, 2024.
  2. Gerstgrasser et al., “Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data”, arXiv:2404.01413, 2024.
  3. Gunasekar et al., “Textbooks Are All You Need”, arXiv:2306.11644, 2023 (phi-1).