Vol. XII · Mid-Training Plate 6 of 10catastrophic forgetting, interference, replay, continual learning, continued pretraining, data mixingLD

Forgetting.

Every weight serves both the old data and the new — so learning the new one undoes the old, unless you keep reminding it.

A neural network has no separate storage for each thing it learns: everything lives in the same weights. Training on a new distribution moves those weights wherever the new data wants them, and whatever the old data needed is overwritten as a side effect — catastrophic interference, first described for neural networks in 1989. How much is lost depends on how far apart the two distributions are and how hard the model is pushed.

The cheapest defence is replay: keep a small share of the original data in every batch of the new phase, so the gradient never forgets entirely where the old optimum was. In continued pretraining of language models a few per cent of replayed data has been found enough to avoid most forgetting. The workshop uses the most distant domain that shares an alphabet: the same computing text with every word spelled backwards, which a bigram model must learn in the same table of weights.

Below: held-out loss on the original text and on the word-reversed text, as the model trains first on one and then on the other, with the chosen share of replayed original data.

What you will learn

Explain why networks forget.
Measure forgetting when the data distribution shifts.
Show how replay prevents it.
Weigh replay’s cost on the new domain.

Workshop

Softmax bigram model, Adam, learning rate 0.05: 1 000 steps on the computing plates, then 1 000 steps on the same text with each word reversed, with a share ρ of each batch drawn from the original text. Held-out loss on 10% of each.

Same weights, two masters

Before the shift the model scores 2.545 nats per character on the original text. Training only on the reversed text for 1 000 steps raises that to 4.031: 1.485 nats of forgetting, most of what it had learned, while the reversed text falls to 2.595. With 10% replay the original text ends at 3.298, preventing 49% of the forgetting; with 5%, 37%; with 30%, 73%. The price is paid on the new text, which ends 0.021 nats worse with 10% replay than without.

Real shifts are milder than reversing every word — plate XII.5’s move from computing to science lost only a few hundredths — but the mechanism is the same, and it scales: a model adapted heavily to code or to a new language without replay loses general ability it will not regain. Other defences exist, such as penalising changes to weights important for the old task, but replay is simple and effective, which is why continued-pretraining recipes nearly always include it.

Forgetting is also why post-training stages (Part Four) mix in pretraining-style data: fine-tuning on a narrow set of instructions can erode knowledge and skills the base model had, the alignment tax of plate XVII.8.

Terms

Catastrophic forgetting: Loss of performance on old data after training on new data.
Interference: Gradients for one distribution undoing what another needed in shared weights.
Replay: Mixing a share of the original data into the new training phase.
Continual learning: Learning a sequence of tasks or distributions without forgetting earlier ones.

Shared weights mean new data overwrites old; a few per cent of replayed original data prevents most of the forgetting.

Sources

McCloskey and Cohen, “Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem”, Psychology of Learning and Motivation 24, 1989.
Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks”, PNAS 2017.
Ibrahim et al., “Simple and Scalable Strategies to Continually Pre-train Large Language Models”, TMLR 2024.