Vol. IV · Attention Plate 1 of 9recurrent neural network, RNN, hidden state, memory, bottleneck, long-range dependenciesA

The Bottleneck.

Read a sentence into a single vector of fixed size and the start of the sentence is gradually written over by the rest.

Before Transformers, the standard neural language model was recurrent: it read one token at a time and kept a single vector, its state, which it updated with each token. Whatever the model needed to remember about the past had to fit in that state. A state of d numbers is the same size after five tokens and after five thousand, so each new token can only be written in by partly overwriting what was there. Information from far back fades.

Gated recurrent networks such as the LSTM (1997) learned to protect important information for longer, and they powered translation and speech systems for years. But the limit was structural. Attention removes it: instead of compressing the past into one vector, the model keeps a vector for every past token and, at each step, looks back at all of them.

Below: a simple memory that reads 40 random token vectors into one state, h ← λh + x, and then tries to recognise each of them from the final state. The curve shows how often the token at each distance back can still be told apart from tokens it never saw; attention, which keeps every token, scores one at every distance.

What you will learn

Describe how a recurrent network carries the past in a fixed-size state.
See how recall of a token decays with its distance.
Relate state size and decay to how much can be remembered.
Explain what attention changes about memory.

Workshop

Recall of the token at each distance from the end of a 40-token sequence, from a state of d numbers that decays by λ per step (blue), against attention, which keeps every token (green).

Writing over the past

With a state of 32 numbers that keeps 80% of itself each step, the token just before the last is recognised 47% of the time, one 8 steps back 5%, and one 16 steps back 0%. Making the state four times larger, 128 numbers, barely helps at distance 8 (5%): after eight steps the old token’s share of the state has decayed to 0.8⁸, about a sixth of a fresh one, and more room does not undo the decay. Slowing the decay to 0.95 keeps old tokens in longer but crowds them together, so even the latest becomes hard to read: recall one step back falls to 13%. A fixed state forces a choice between remembering long and remembering clearly.

Real recurrent networks learned much better update rules than this one, and gating let an LSTM carry a bit of information across hundreds of steps. But the arithmetic stays: everything from the past must pass through d numbers. Attention changes the arithmetic. It stores a key and a value for every token (40 × d numbers here) and lets each new token look up whichever of them it needs. Memory grows with the text, and so does the cost (plate IV.7) — which is why state-space models, a modern take on recurrence, are back on the research agenda (plate VII.8).

The original sequence-to-sequence translators of 2014 compressed a whole source sentence into one vector before decoding it; their quality fell off sharply for sentences longer than about twenty words — the observation that led Bahdanau, Cho and Bengio to add attention.

Terms

Recurrent neural network: A network that reads one token at a time, updating a fixed-size state.
Hidden state: The vector a recurrent network carries from step to step; its only memory.
LSTM: Long short-term memory (1997): a recurrent network with gates that control what is kept and forgotten.
Bottleneck: A fixed-size channel through which all information must pass.

A recurrent model must squeeze the whole past into a fixed-size state, so distant tokens fade. Attention keeps every token and looks back at all of them.