Vol. XXIII · Serving Plate 2 of 9KV cache, keys, values, memory, grouped-query attention, multi-head latent attention, context lengthCE

The KV Cache.

Keep every past token’s keys and values. It saves the recomputation and spends the memory.

A new token attends to every earlier token’s key and value vectors in every layer (plate IV.2). Those vectors do not change once computed — a causal model never revises the past — so servers keep them: the key–value cache. Without it, generating each new token would mean running the whole prefix through the model again; with it, each step processes one token and reads the stored vectors.

The price is memory. Per token the cache holds a key and a value for each key–value head in each layer: 2 × layers × KV heads × head size × bytes. Architecture decides how much: multi-head attention keeps one key–value pair per query head; grouped-query attention (plate VII.2) shares each pair among a group of query heads; DeepSeek’s multi-head latent attention stores one compressed vector per layer. The workshop computes the cache of published configurations and how many sequences of a given length fit beside Llama 3 8B on an 80 GB GPU.

Below: cache bytes per token for five published models on a logarithmic scale; readouts give the cache for one sequence of the chosen length and how many such sequences fit.

What you will learn
Workshop
BF16 (2 bytes). GPT-3 175B: 96 layers, 96 heads of 128 (multi-head). Llama 3 8B, 70B and 405B: 32, 80 and 126 layers, 8 key–value heads of 128 (grouped-query). DeepSeek-V3: 61 layers, a 512-dimensional compressed latent plus a 64-dimensional positional key per layer (multi-head latent attention). Sequences that fit: 80 GB × 90% minus Llama 3 8B’s 16.1 GB of weights, divided by one sequence’s cache.

Memory that grows with every token

Llama 3 8B stores 131.1 KB of keys and values per token, so one 8 192-token conversation holds 1.1 GB and a full 131 072-token context 17.2 GB — more than the model’s own weights. After the weights, an 80 GB GPU has room for about 52 conversations of 8 192 tokens. GPT-3’s multi-head attention needed 36 times as much per token as Llama 3 8B; DeepSeek-V3’s latent attention needs 1.9 times less than Llama 3 8B despite being a far larger model.

Recomputing instead would cost the whole prefix’s forward pass at every step: generating 500 tokens after a 1 000-token prompt would process about 1,250 tokens per output token on average instead of one. No production server does that. The cache instead makes memory the resource that limits how many users a GPU serves at once, which is why the attention variants of Vol. VI were invented, why caches are quantised (plate XXIV.3), and why their allocation is the subject of the next plate.

Grouped-query and latent attention were designed for inference: they cost little quality in training and cut the per-token cache that sets serving concurrency.

Terms

KV cache
Stored keys and values of past tokens for every layer.
Grouped-query attention
Sharing each key–value head among several query heads.
Multi-head latent attention
Caching one compressed vector per layer instead of keys and values.
Concurrency
How many sequences are served at once.
The KV cache trades recomputation for memory; its size per token, set by architecture, decides how many users fit on a GPU.

Sources

  1. Brown et al., NeurIPS 2020, Table 2.1 (GPT-3).
  2. Llama Team, AI @ Meta, arXiv:2407.21783, 2024, Table 3.
  3. DeepSeek-AI, “DeepSeek-V3 Technical Report”, arXiv:2412.19437, 2024, section 2.1.1.
  4. Ainslie et al., “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints”, EMNLP 2023.