Vol. XXIV · Making Inference Cheap Plate 3 of 10KV cache quantisation, FP8 cache, KIVI, keys, values, concurrency, memoryAE

Quantising the Cache.

The cache is the memory that grows. Store it in fewer bits and more users fit.

Weights are read once per step for everyone; each sequence’s cache is read once per step for that sequence alone, and at long contexts the caches outweigh the weights (plate XXIII.5). Quantising the cache attacks that term directly. Servers commonly store it in 8-bit floating point; research methods go further — KIVI (Liu et al., 2024) reaches 2 bits by quantising keys per channel and values per token, because keys have a few channels with large values and values do not.

The course model micro (1.8 million parameters, four layers; plate I.9) is quantised, pruned and sped up in the page, and its loss is measured on held-out course text it never trained on. The workshop rounds every key and value micro computes, per head and per token, to the chosen bit width before it enters the cache, and measures the held-out loss; it also computes what the same width would mean for Llama 3 8B on an 80 GB GPU.

Below: left, micro’s loss increase for each cache bit width; right, how many 8 192-token conversations fit beside Llama 3 8B on an 80 GB GPU at that width.

What you will learn
Workshop
micro: symmetric round-to-nearest of each head’s 32-dimensional key and value at every token, as they are cached; queries and weights untouched. Loss on the first 1 500 characters of the English and 500 of the Chinese held-out plates. Concurrency: Llama 3 8B with 16.1 GB of BF16 weights on 80 GB at 90% use, 8 192-token conversations, cache scales ignored.

Twice the users for nearly nothing

An 8-bit cache raises micro’s loss by 0.0008 nats per token, and a 4-bit cache by 0.044; at 3 bits the increase is 0.238 and at 2 bits 0.96, where the rounded keys no longer pick out the right positions to attend to. For Llama 3 8B the payoff is direct: 52 conversations of 8 192 tokens fit with a 16-bit cache, 104 with 8 bits and 208 with 4 — and every decode step reads proportionally fewer bytes of cache.

The simple per-head, per-token rounding here is the bluntest scheme. Keys are harder than values because a few channels of every key carry large values — the same outliers that trouble activation quantisation — so methods like KIVI quantise keys per channel instead, and keep the most recent tokens in full precision until a group fills. With such care, published results reach 2 to 3 bits with small accuracy loss; 8-bit floating point is what production servers commonly enable.

Cache quantisation combines with the architectural savings of plate XXIII.2: grouped-query or latent attention shrinks how many numbers are cached, quantisation how many bits each takes.

Terms

KV cache quantisation
Storing cached keys and values in fewer bits.
Per-token
One scale for all of a token’s values in a head.
Per-channel
One scale for one dimension across many tokens.
Concurrency
Sequences served at once, limited by cache memory.
Quantising the cache multiplies the users a GPU holds; 8 bits is nearly free, and lower widths need schemes that respect the keys’ outlier channels.

Sources

  1. Liu et al., “KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache”, ICML 2024.
  2. Hooper et al., “KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization”, NeurIPS 2024.
  3. Kwon et al., SOSP 2023.