Vol. VII · Modern Architectures Plate 3 of 10multi-head latent attention, MLA, KV compression, low rank, DeepSeek, decoupled RoPEAC

Latent Attention.

Store a compressed summary of each token’s keys and values, and rebuild them for every head when needed.

Grouped-query attention shrinks the cache by throwing away key–value heads. Multi-head latent attention (MLA), introduced by DeepSeek in 2024, keeps all the heads but caches far less: each token’s hidden vector is projected down to a small latent vector — 512 numbers in DeepSeek-V3 — and that latent is all that is stored. Every head’s keys and values are up-projections of it, and the up-projection matrices can be folded into the query and output projections so that they are never materialised during generation.

Rotary positions complicate this, because a rotation applied after the up-projection would break the folding. MLA therefore keeps a small separate key part — 64 numbers per token in DeepSeek-V3 — that carries RoPE and is shared across heads. The cache per token per layer is then 512 + 64 numbers. The workshop compares that with standard, grouped and multi-query attention for DeepSeek-V3’s published shape.

Below: the key–value cache per token of DeepSeek-V3’s published shape (61 layers, 128 heads of 128 dimensions) under four attention schemes, and the cache for one sequence at the chosen context length, in 16-bit numbers.

What you will learn
Workshop
Cache per token for DeepSeek-V3’s published shape under multi-head (MHA), grouped-query with 8 key–value heads (GQA), multi-query (MQA) and latent attention (MLA: 512-number latent plus 64-number rotary key), in 16-bit.

Compress, then rebuild

For DeepSeek-V3’s published shape, standard multi-head attention would cache 4.0 MB per token; MLA caches 70.3 KB, about 57 times less — and even 3.6 times less than an 8-head grouped-query design would. At 131 072 tokens one sequence’s cache is 9.2 GB instead of 524.0 GB. The DeepSeek-V2 paper reported a 93.3% reduction in cache against the company’s earlier dense 67-billion-parameter model, with no loss in quality on its benchmarks.

The price is complexity and some compute. Keys and values are reconstructed from the latent, which is a low-rank constraint the model must learn to live with; the decoupled rotary part adds bookkeeping; and inference kernels must fold the up-projections into neighbouring matrices to avoid ever building full per-head keys. Whether latent attention or aggressive grouping is the better trade is a live design question; what the arithmetic shows is how much of a long-context serving bill is attention’s cache, and how far it can be cut.

The same low-rank trick is applied to queries in DeepSeek’s models, not to save cache — queries are never cached — but to save activation memory during training.

Terms

Multi-head latent attention (MLA)
Caching a small latent vector per token and reconstructing every head’s keys and values from it.
Latent vector
A compressed representation, here of a token’s keys and values.
Low-rank
A matrix that factors through a smaller dimension; keys and values limited to that subspace.
Decoupled RoPE
A small separate key part carrying rotary positions, outside the compression.
MLA caches one small latent per token instead of per-head keys and values; for DeepSeek-V3’s shape that is about 57 times less cache than standard attention.

Sources

  1. DeepSeek-AI, “DeepSeek-V2”, arXiv:2405.04434, 2024 (MLA introduced; 93.3% KV-cache reduction against DeepSeek 67B).
  2. DeepSeek-AI, “DeepSeek-V3 Technical Report”, arXiv:2412.19437, 2024 (configuration).