Vol. XXX · The Research Frontier Plate 6 of 11data wall, data constraints, human-generated text, scaling, repetition, synthetic dataCE

Running Out of Data.

Compute keeps growing; the internet does not grow as fast.

This volume describes the field as of October 2026. Claims are marked established (replicated or long-standing), reported (from a single lab or report, not independently reproduced) or contested (actively disputed). Villalobos and colleagues (ICML 2024) estimated the effective stock of public human-generated text at about 300 trillion tokens (90% interval 100 trillion to 1000 trillion) and projected that frontier training would use it up between 2026 and 2032 (reported estimate with wide uncertainty). Compute-optimal training (plate X.3) wants about 20 tokens per parameter, so data needed grows with the square root of compute — and frontier compute has been growing several-fold a year.

Muennighoff and colleagues found up to 4 epochs of repeated data nearly as good as fresh data (established for their settings), which stretches the stock. The workshop projects the tokens a compute-optimal frontier run would need, from Llama 3 405B’s 3.8 × 10²⁵ FLOPs in 2024, at the chosen compute growth per year, and finds when it exceeds the stock times the allowed repetition.

Below: tokens wanted by a compute-optimal frontier run each year against the usable stock; readouts give the crossing year.

What you will learn
Workshop
C₂₀₂₄ = 3.8 × 10²⁵ FLOPs (Llama 3 405B); compute-optimal tokens D = √(20C/6) (C = 6ND, D = 20N); stock S from Villalobos et al. (central 300 trillion tokens); e = epochs of repetition allowed. Years before 2024 not shown. A projection under stated assumptions, not a forecast.

A wall that moves with assumptions

A compute-optimal run at Llama 3 405B’s 2024 compute wants about 11.3 trillion tokens. At four-fold compute growth a year, it outgrows a single pass over the central 300-trillion-token stock around 2033 and four passes around 2037. Every assumption moves the date by years — the stock’s uncertainty spans a factor of ten, growth rates vary, and frontier runs already train far past 20 tokens per parameter, which wants more data, not less. The direction is robust; the date is not.

The responses each have a status. Repetition is established up to a few epochs. Synthetic data is in heavy use and its ceiling is contested (previous plate). Multimodal data — video, audio — adds tokens of uncertain value for language (contested). Reinforcement learning extracts more from fewer tokens and has absorbed a growing share of compute (reported shift). And private data — licensed books, enterprise documents, user interactions — raises legal and consent questions (plate IX.9). The wall is less a cliff than a rising price per useful token.

Data that has already been used is not lost — the stock estimate counts tokens that exist, not tokens that remain — but each additional epoch over the same text is worth less than fresh text.

Terms

Data wall
The point where wanted training tokens exceed usable human text.
Stock of text
The estimated quantity of usable public human-written text.
Repetition
Training several epochs over the same data.
Compute-optimal tokens
About 20 per parameter at the Chinchilla optimum.
Human-written text is finite while wanted tokens grow with compute; projections put the meeting within years, with every assumption moving the date.

Sources

  1. Villalobos et al., “Will we run out of data? Limits of LLM scaling based on human-generated data”, ICML 2024.
  2. Muennighoff et al., “Scaling Data-Constrained Language Models”, NeurIPS 2023.
  3. Hoffmann et al., “Training Compute-Optimal Large Language Models”, NeurIPS 2022.