Vol. VIII · The Pretraining Run Plate 7 of 10memorisation, canaries, duplication, extraction, privacy, exposure, CarliniLE

Memorisation.

Plant a secret in the training data a known number of times — and see how many copies it takes before the model can recite it.

Language models learn general patterns, but they can also store particular strings verbatim: a licence text, a famous quotation, or a phone number that appeared in a scraped page. Carlini and colleagues showed in 2021 that GPT-2 could be made to emit memorised personal information, and in 2023 that memorisation grows with model size, with the number of times a string is duplicated in the data, and with the length of the prompt that cues it.

The cleanest way to measure it is with canaries: random strings with no pattern to learn, planted in the training data a controlled number of times. Two recorded runs did this. One hundred sentences of the form “The access code for room N is d d d d.”, each with a random room and four random digits, were spliced into the corpus — twenty sentences at each of 1, 4, 16, 64 and 256 copies — and the trained models were asked to complete each prefix.

Below: left, the share of canaries whose four-digit code the model reproduces exactly from the prefix, by number of copies, for a micro-sized and a mini-sized model. Right, the code’s loss in nats against a random guess.

What you will learn
Workshop
Recorded runs with planted canaries; 20 canaries at each duplication count; each run reads its data about three times, so each copy is seen about three times. Left: share of canaries extracted by greedy decoding. Right: mean loss of the code (four digits and the full stop) given the prefix; dashed, the loss of guessing four digits at random.

Copies make memories

The micro-sized model never recites a code it saw once, four or sixteen times. For codes planted once, the code’s loss is 13.6 nats — worse than the 9.2 nats of guessing four digits at random, because the model expects ordinary words, not digits, after “is”. With more copies the loss falls (10.0 nats at 16, 4.9 at 64) before exact recital appears: 5% of the codes at 64 copies, 90% at 256, where the code costs only 1.49 nats. Recital is the all-or-nothing end of a gradual process — the same shape as plate VIII.6.

The mini-sized model memorises sooner: its loss on codes seen 16 times is already 6.7 nats, and it recites 85% of the codes planted 64 times, where the smaller model recites 5%. That is Carlini’s finding in miniature — memorisation grows with duplication and with model size — and it is the strongest argument for deduplication (plate IX.4): a string that appears hundreds of times in a corpus, such as a licence, a boilerplate disclaimer or a leaked record copied across sites, is exactly what a large model will reproduce. Each copy in these runs is seen about three times, since each run reads its data about three times.

Memorisation is not always bad: a model should know the text of a famous poem or the digits of π that appear everywhere. The trouble is that the same mechanism stores a private address that appeared on ten scraped pages.

Terms

Memorisation
Reproducing a training string verbatim, beyond what its patterns predict.
Canary
A random string planted in training data to measure memorisation.
Duplication
The number of times a string appears in the training data.
Extraction
Recovering a memorised string by prompting the model.
Memorisation rises with duplication and with model size — the strongest reasons to deduplicate training data.

Sources

  1. Carlini et al., “Extracting Training Data from Large Language Models”, USENIX Security 2021.
  2. Carlini et al., “Quantifying Memorization Across Neural Language Models”, ICLR 2023.
  3. Carlini et al., “The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks”, USENIX Security 2019 (canaries).