Vol. XII · Mid-Training Plate 9 of 10knowledge cutoff, training data date, release lag, temporal drift, effective cutoff, stalenessDE

The Knowledge Cutoff.

Training data stops at some date; release comes months later; use goes on for years — and the model cannot see any of that from inside.

Every model has a knowledge cutoff: the date of the newest text in its training data. Model cards state it. Meta’s card for Llama 3, released on 18 April 2024, gives March 2023 for the 8B model and December 2023 for the 70B; Llama 3.1, released in July 2024, gives December 2023. Between the cutoff and release lie months of training, mid-training, post-training and evaluation, and after release a model is used for a year or more.

The stated cutoff overstates what the model knows about its final months. Web pages about recent events keep being written for years, so a crawl taken soon after an event holds only a thin early slice of what will eventually exist; and crawls deduplicate against older copies of pages. Cheng and colleagues (2024) found that models’ effective cutoffs — where their knowledge actually thins out — are often well before the stated ones. Without a date in its prompt, a model also has no way to know how much time has passed since it was trained.

Below: left, stated cutoffs and release dates on one time axis. Right, for the selected model, the months from cutoff to release, and from cutoff to a day of use you choose.

What you will learn
Workshop
Stated knowledge cutoffs and release dates from Meta’s model cards: Llama 3 8B (cutoff March 2023), Llama 3 70B (December 2023), both released 18 April 2024; Llama 3.1 (cutoff December 2023, released 23 July 2024). Right: months from the selected model’s cutoff to its release and to the chosen date of use.

Stale from the day of release

Llama 3 8B’s stated knowledge ends 14 months before its release, the 70B’s 5 months before, and Llama 3.1’s 8. Used in mid-2025, the 8B model is working from a picture of the world more than two years old, and nothing inside it says so: a model has no clock, and unless the date is put in its prompt it will tend to assume that the latest events it read about are recent. Lazaridou and colleagues showed that language models’ predictions degrade steadily on text written after their training data ends, and the degradation grows with the gap.

Effective cutoffs make the gap larger. Text about any month keeps being written for years, so the final months before a cutoff are thinly represented; Cheng and colleagues found effective cutoffs often well before the stated ones, partly because deduplication and crawl timing keep older versions of pages. The remedies are outside the weights: put the current date and relevant documents in the context (retrieval, plate XXV.3), or retrain and continue-pretrain on newer data (plate XII.5) — each with the costs this volume has described.

The cutoff matters for evaluation too: a benchmark written after a model’s cutoff cannot have been in its training data, which makes post-cutoff test sets a standard defence against contamination (plate IX.10).

Terms

Knowledge cutoff
The date of the newest text in a model’s training data.
Effective cutoff
Where a model’s knowledge actually thins out, often earlier than stated.
Release lag
The time from cutoff to public release.
Temporal drift
The growing mismatch between a model’s training data and the world as time passes.
A model’s knowledge ends before its release and ages throughout its use; its effective cutoff is earlier than stated, and it cannot tell the date by itself.

Sources

  1. Meta, Llama 3 model card (April 2024) and Llama 3.1 model card (July 2024).
  2. Cheng et al., “Dated Data: Tracing Knowledge Cutoffs in Large Language Models”, COLM 2024.
  3. Lazaridou et al., “Mind the Gap: Assessing Temporal Generalization in Neural Language Models”, NeurIPS 2021.