Vol. IX · Data Plate 10 of 12contamination, benchmark leakage, n-gram overlap, 13-gram, test set, memorisation, evaluation validityDE

Contamination.

A test the model has already read measures memory, not ability — and an n-gram check only finds the copies that were not changed.

Benchmarks are published on the web, discussed on forums, copied into study guides and answer sites, and so they end up in crawls. A model that has read a test item, perhaps many times, can answer it from memory; its score then measures exposure, not ability. GPT-3’s report checked for this by looking for 13-word sequences shared between each test item and the training data and reporting scores on the clean subset separately. Most reports since include some version of the check.

An n-gram check is simple and has a known blind spot. Long n-grams make false alarms rare, because a 13-word sequence seldom repeats by chance; but they miss any copy that was lightly edited — reformatted, reworded, or with numbers changed — since a single changed word breaks every n-gram that contains it. The workshop builds a toy benchmark from course text: sentences copied verbatim from a training part, sentences with some words changed, and sentences from text the training part does not contain.

Below: left, the share of each test item’s n-grams found in the training part, for verbatim copies, edited copies and clean items. Right, how many of each a 50% overlap threshold flags.

What you will learn
Workshop
The first 60% of the course slice is the toy training set. Test items: 15 sentences copied from it, 15 other training sentences with each word changed with the chosen probability, and 15 sentences from the remaining 40%. Left: mean overlap of word n-grams with the training set. Right: items flagged at a 50% overlap threshold.

Finding copies that were not changed

With 13-grams — the length GPT-3 used — every verbatim copy is flagged (15 of 15) and no clean item is (0). Edit one word in ten and the check catches only 5 of the 15 edited copies, whose mean overlap falls to 35%; with a fifth of the words changed it catches 2. Shorter n-grams are more forgiving of edits — with 5-grams, 10 of the 15 lightly edited copies are flagged — but on real data short n-grams also match common phrases in unrelated text, so false alarms rise.

Because the check misses rewrites, the stronger evidence for contamination is behavioural. One approach writes a fresh test of the same difficulty and compares: Zhang and colleagues built GSM1k to mirror the GSM8K arithmetic benchmark and found that some models scored markedly lower on the new problems, consistent with overfitting to the published set. Others test whether a model can complete a benchmark item verbatim from its first half, or score items it could not have seen because they were written after its cutoff. Evaluation returns to this in Vol. XXI.

Contamination is not always accidental. A benchmark’s answers can leak through synthetic data generated by a model that had seen them, or through fine-tuning sets assembled from public sources — which is why held-out, regularly refreshed test sets have become common.

Terms

Contamination
Test items, or close copies, present in the training data.
n-gram overlap
The share of an item’s n-word sequences found in the corpus.
Clean subset
Test items with no detected overlap, scored separately.
Fresh test
A newly written test of the same kind, which the model cannot have seen.
Contamination turns a test into a memory check; n-gram overlap finds verbatim copies and misses edited ones.

Sources

  1. Brown et al., “Language Models are Few-Shot Learners”, NeurIPS 2020, Appendix C.
  2. Sainz et al., “NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark”, Findings of EMNLP 2023.
  3. Zhang et al., “A Careful Examination of Large Language Model Performance on Grade School Arithmetic”, arXiv:2405.00332, 2024 (GSM1k).