Drills: Serving.
Five questions at a time from the whole volume, until every serving number has a mechanism behind it.
This plate draws rounds of five from the questions and predictions of the eight plates before it, with the reasoning and source plate after each answer.
The volume’s thread is that decode reads the weights for every token, so serving is the art of filling each read with many users — limited by the memory their caches need, scheduled step by step, and paid for by the GPU-hour.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Explain prefill and decode bottlenecks.
- Size KV caches.
- Reason about batching, latency and throughput.
- Estimate the cost of a token.
What to carry forward
Prefill is compute-bound and decode memory-bound (plate XXIII.1); the KV cache trades recomputation for memory set by architecture (plate XXIII.2); paging stops that memory being wasted (plate XXIII.3); continuous batching fills the GPU without making requests wait for each other (plate XXIII.4).
Batch size is the dial between latency and throughput (plate XXIII.5); shared prefixes are prefilled once (plate XXIII.6); disaggregation separates the phases at the cost of moving caches (plate XXIII.7); and cost is price per GPU-hour over tokens per hour (plate XXIII.8). Vol. XXIV makes each token cheaper by shrinking what has to be read.
Terms
- Prefill
- The prompt in one pass.
- Decode
- One token per pass.
- Paged attention
- Cache allocated in blocks.
- TPOT
- Time per output token.