Vol. XXX · The Research Frontier Plate 3 of 11memory, continual learning, catastrophic forgetting, long context, retrieval, external memoryCE

Memory Beyond the Context.

The weights stop learning at release; the context forgets at the end of the session. Everything else is engineering.

A deployed model has two memories. Its weights hold what it learned in training and change only when it is retrained. Its context holds the current conversation and is gone when the session ends. Products that remember users across sessions, and agents that work for days, add a third: external memory — notes the model writes, a retrieval index of past interactions, summaries carried forward (plates XXV.3 and XXV.4). Context windows grew from thousands to millions of tokens in two years (established), but cost per turn grows with what is kept (plate XXIII.5).

The ideal — a model that keeps learning from experience, updating its weights without losing what it knew — is continual learning, and its central obstacle, catastrophic forgetting, has been known since 1989. The workshop compares three ways to carry M tokens of memory through a session of T turns: keep it all in context, retrieve k relevant chunks each turn, or fine-tune it into the weights between sessions.

Below: tokens processed per session for each memory strategy as the memory grows; readouts at the chosen size.

What you will learn
Workshop
A session of T = 50 turns with q = 500 new tokens per turn. In context: all M memory tokens re-read every turn. Retrieval: k = 5 chunks of c = 400 tokens per turn plus a one-off indexing pass over M. Weights: no memory tokens per turn, plus a fine-tuning pass costed as 3 × M tokens (forward and backward), with the forgetting risk not costed. A cost model, not a measurement.

Engineering around an unsolved problem

With 100 000 tokens of memory, keeping it all in context costs 5.03 million tokens over a 50-turn session, retrieval about 225 thousand — 22 times less — and fine-tuning less still per turn, if forgetting could be ignored. At ten million tokens, context is out of the question (500 million) while retrieval stays near the cost of indexing (10.1 million). That is why deployed memory is mostly retrieval and summaries: it scales, it can be inspected and deleted, and it does not risk the model’s other abilities.

Writing experience into weights is the open problem. Fine-tuning on new material degrades old skills unless old data is replayed or the update is constrained, and doing this continually, safely and cheaply for every user is unsolved. Research directions in 2026 include memory layers and test-time training (models that update some parameters while reading), learned memory management, and agents that curate their own notes; none is yet established as a replacement for retrieval plus context.

External memory has a privacy dimension weights do not: notes can be shown to the user and deleted, whereas something learned into weights cannot be cleanly removed — the subject of machine unlearning.

Terms

Continual learning
Learning from new data over time without forgetting old skills.
Catastrophic forgetting
Loss of earlier abilities when training on new data.
External memory
Stored notes or indexes the model reads and writes outside its weights.
Test-time training
Updating some parameters while processing an input.
Models remember across sessions through external memory, because learning continually in the weights without forgetting is still unsolved.

Sources

  1. McCloskey and Cohen, “Catastrophic Interference in Connectionist Networks”, 1989.
  2. Kirkpatrick et al., “Overcoming catastrophic forgetting in neural networks”, PNAS 2017.
  3. Packer et al., “MemGPT”, arXiv:2310.08560, 2023.
  4. Gemini Team, “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context”, arXiv:2403.05530, 2024.