Vol. VI · Positions and Long Context Plate 7 of 10sliding-window attention, attention sinks, StreamingLLM, local attention, memoryAC

Windows and Sinks.

Let each token look back only a fixed distance — but keep the first few tokens in view, or the model loses its footing.

One way to bound attention’s cost is to stop each token looking all the way back: with a sliding window of W tokens, a query attends only to the most recent W keys, so the work per token and the key–value cache are fixed whatever the length. Information from further back is not lost entirely — in a deep model each layer’s window starts from tokens that already gathered information from their own windows — but it arrives indirectly. Mistral 7B (2023) used a 4 096-token window in every layer; later models such as Gemma mix windowed and full-attention layers.

A surprise came from streaming: when the oldest tokens were evicted from a model’s window, quality collapsed even though they were far away. Xiao and colleagues (2023) traced it to attention sinks — models dump unneeded attention on the first few tokens (plate IV.6) — and showed that keeping just those few, plus the window, restores almost full quality. The workshop measures both effects on micro.

Below: micro’s average loss over 128 tokens of course text when attention is restricted to the last W tokens, with no sink tokens kept, with the first token kept, and with the first four kept; the dashed line is full attention.

What you will learn
Workshop
micro’s mean loss on 128 tokens of held-out course text with attention limited to the last W positions: no sink (red), first token kept (ochre), first four kept (green); dashed: full attention.

The first tokens matter

With full attention micro averages 4.182 nats on this English passage. Restrict every token to the last 16 and it rises to 4.603; to the last 8, 4.830. Now keep the first token in view as well: at window 16 the loss falls to 4.344, and keeping the first four gives 4.264, close to full attention. The first tokens of the text carry little information about the words near the end, yet removing them costs more than removing many nearer tokens — because micro’s sink heads (plate IV.6) have learned to put spare attention there, and without them that attention lands on tokens it disturbs.

The same pattern was found in models with billions of parameters, and it led to a cheap engineering fix for streaming generation: keep a handful of initial tokens forever and a sliding window of recent ones, and a model can generate indefinitely with a fixed cache. Windows also save memory in serving (Vol. XXIII), which is why architectures mix them with full-attention layers. What a window cannot do is retrieve a specific fact from far outside it; long-context retrieval is measured next.

Some models now add a learnable “register” or sink token at the start of every sequence, or a learned bias in the softmax, so that the parking place for spare attention is designed rather than improvised on whatever token happens to come first.

Terms

Sliding-window attention
Each token attends only to the most recent W tokens.
Attention sink
An early token, usually the first, on which heads park attention they do not need.
Streaming generation
Generating indefinitely with a fixed-size cache by evicting old tokens.
Local and global layers
Mixing windowed attention layers with full-attention ones.
A sliding window bounds attention’s cost, but the first few tokens must stay in view: models use them as sinks, and evicting them hurts more than evicting many recent tokens.

Sources

  1. Jiang et al., “Mistral 7B”, arXiv:2310.06825, 2023.
  2. Xiao, Tian, Chen, Han and Lewis, “Efficient Streaming Language Models with Attention Sinks”, ICLR 2024 (arXiv 2023).