Vol. XXIII · Serving Plate 6 of 9prefix caching, prompt caching, RadixAttention, system prompt, KV cache reuse, time to first token, pricingCE

Reusing Prefixes.

If a thousand requests start with the same two thousand tokens, compute those tokens once.

The KV cache of a token depends only on the tokens before it. Two requests that share their first 1 500 tokens therefore share those tokens’ keys and values exactly, and a server that kept them from the first request can start the second’s prefill at token 1 501. Shared prefixes are everywhere: a product’s system prompt on every request, a long document asked several questions, each turn of a chat resending the whole conversation, an agent’s growing transcript.

Prefix caching keeps recently used prefixes’ blocks (plate XXIII.3) and matches new requests against them — SGLang’s RadixAttention keeps them in a radix tree of token sequences, vLLM hashes blocks. API providers pass the saving on, billing cached input tokens at a fraction of the normal price. The workshop serves Llama 3 8B with 2 048-token prompts of which the first 1 536 are a shared system prompt.

Below: left, median and 95th-percentile time to first token and median time per output token with and without the prefix cache, at the chosen rate; right, the 95th-percentile time per token as the rate rises.

What you will learn
Workshop
Simulated: Llama 3 8B in BF16 on one H100, Poisson arrivals, prompts exponential with mean 2 048 tokens of which the first 1 536 are shared, outputs mean 256, continuous batching, paged cache. With the cache, the shared tokens are never prefilled again. Times are roofline estimates from the H100’s published specifications (989 TFLOP/s dense BF16, 3.35 TB/s), assuming 60% of peak compute and 80% of peak bandwidth are reached plus 0.2 ms per pass — not measurements.

Compute the shared part once

At 12 requests a second, the prefix cache cuts the median time to first token from 0.119 s to 0.021 s: 51% of all prompt tokens are never prefilled again. The decode side benefits too, because every prefill skipped is a stall the running users do not see — the 95th-percentile time per token falls from 47.8 ms to 16.0 ms. At 16 requests a second the uncached server is overloaded, with a 95th-percentile first-token wait of 8.7 s against 0.15 s cached.

The saving depends on traffic that actually repeats, and on the cache surviving until the next request: servers evict least-recently-used prefixes when memory is short, and providers keep cached prompts for minutes, not days. Prompt design follows: put the stable material — instructions, tools, documents — first and the variable material last, because a single changed token early in the prompt invalidates everything after it. Anthropic charges 10% of the input price for cached tokens, a direct measure of how much cheaper skipped prefill is.

Caching works only on exact prefixes: the same text in a different position, or after a different first token, has different keys and values.

Terms

Prefix caching
Reusing the KV cache of a shared beginning of prompts.
RadixAttention
SGLang’s prefix cache, organised as a radix tree of token sequences.
Cache hit
A request whose prefix is already cached.
Eviction
Discarding cached prefixes to free memory, usually least recently used first.
Prefixes shared across requests need prefilling only once; caching them shortens waits, steadies decoding and cuts the price of input.

Sources

  1. Zheng et al., “SGLang: Efficient Execution of Structured Language Model Programs”, NeurIPS 2024 (RadixAttention).
  2. Kwon et al., SOSP 2023 (block sharing).
  3. Anthropic, “Prompt caching” documentation (accessed October 2026).