Vol. XXI · Evaluation Plate 7 of 10benchmark saturation, contamination, leakage, held-out, canary, private test sets, dynamic benchmarksEL

Saturation and Leakage.

A score near the ceiling says nothing. A score on questions the model has seen says less.

Benchmarks have short lives. When most models score near the maximum — as happened to GLUE, then SuperGLUE, then MMLU and GSM8K within a few years each — the remaining differences are noise and errors in the answer key, and the field moves to harder tests. Worse is contamination: benchmark questions, often with answers, appear on the web and enter pre-training or fine-tuning data (plate IX.10), and a score then measures memory rather than skill. Contamination can be partial, indirect (paraphrases, solutions in forums) and invisible without the training data.

The workshop measures it directly. The 30 held-out sums are split into a “public” half and a “private” half; the fine-tuned model is trained as in plate XVII.1, but with the chosen number of public questions and their answers mixed into its fine-tuning data. Its score on the public half is the reported benchmark; the private half shows what it can actually do.

Below: accuracy on the public half and on the private half of the held-out benchmark, against how many public questions leaked into fine-tuning.

What you will learn
Workshop
The fine-tuned model of plate XVII.1 (30 human examples, 150 steps), retrained with 0 to 15 of the public half’s questions and answers added to its fine-tuning data. Public: the first 15 held-out sums; private: the other 15, never trained on. Greedy answers.

The score on what it has seen

Without leakage the model scores 73% on the public half. With all 15 public questions and answers in its fine-tuning data, it scores 87%: a gain of 13% on the benchmark that reflects memory, not arithmetic, while the private half — the same kind of question, unseen — scores 40% and shows no consistent gain across the leak levels. Note that the two halves differ even without leakage: 15 questions each is a small sample, and the halves differ in difficulty, which is exactly why a contamination check compares like with like.

Real contamination is rarely so clean. Questions leak with paraphrases, solutions appear in forums and textbooks, and a model may have seen a benchmark’s questions without answers. Detection relies on the training data where it is known (n-gram overlap, plate IX.10), canary strings embedded in benchmark files so their inclusion can be detected, perturbed re-tests — GSM1k rewrote grade-school maths problems in the style of GSM8K and found some models scored markedly lower on the fresh versions — and private or continually refreshed test sets that never appear online.

Saturation and contamination are why evaluation is a moving target: the most informative benchmarks are usually the newest, before they are either solved or absorbed into training data.

Terms

Saturation
Most models scoring near a benchmark’s maximum.
Contamination
Benchmark items present in a model’s training data.
Canary string
A unique marker in benchmark files, to detect their inclusion in training data.
Private test set
Evaluation items kept off the public web.
A score on leaked questions measures memory; compare against equivalent unseen items, keep private sets, and prefer fresh benchmarks.

Sources

  1. Kiela et al., “Dynabench: Rethinking Benchmarking in NLP”, NAACL 2021.
  2. Sainz et al., “NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark”, Findings of EMNLP 2023.
  3. Zhang et al., “A Careful Examination of Large Language Model Performance on Grade School Arithmetic”, NeurIPS 2024 (GSM1k).