Vol. XXXI · History and Projects Plate 12 of 14project, serving, continuous batching, paged attention, quantisation, speculative decoding, latency, costCE

Project: Serve a Model.

Latency targets, a cost ceiling and a quality floor: choose the batching, the cache, the precision and the speculation.

The brief: Llama 3 8B serving 16 requests a second, prompts averaging 1 024 tokens and outputs 256, on H100s at an assumed $2 per GPU-hour. The 95th-percentile time to first token must be at most 0.5 s, the 95th-percentile time per output token at most 15 ms, and the cost at most $0.20 per million generated tokens; and the quality team forbids weights below 8 bits (plates XXIV.1–2).

Five levers from Vols. XXIII–XXIV: static or continuous batching, a contiguous or paged KV cache, one or two GPUs, 16-, 8- or 4-bit weights, and speculative decoding with a draft that proposes 0, 2 or 4 tokens, each accepted with probability 0.7, at a tenth of a target pass per drafted token. The starting server — static batching, contiguous cache, one GPU, 16-bit, no speculation — has a 95th-percentile time to first token of 73.9 s. A project plate grades the design against a brief: every check below the workshop must pass, and a passing design is saved in this browser.

Left: the three latency and cost measures against their limits, as a share of each limit. Right: throughput and GPU memory left for the cache.

What you will learn
Workshop
The serving simulator of Vol. XXIII: 60 simulated seconds of Poisson arrivals, exponential prompt and output lengths, prefill and decode passes timed by the roofline. Times are roofline estimates from the H100’s published specifications, assuming 60% of peak compute and 80% of peak bandwidth plus 0.2 ms per pass — not measurements. Quantised weights shrink the bytes read per pass; their effect on quality is not simulated, hence the floor.

Throughput first, then the per-token time

The starting server fails on queueing, not on speed: static batching makes every new request wait for the current batch to finish, and contiguous caches reserve the maximum length for every sequence, so few fit. Continuous batching and paging fix the time to first token and the cost, but with many sequences in each decode step, the 95th-percentile time per token stays above 15 ms. Two GPUs would fix that at twice the cost, over the ceiling.

What is left is the per-token time, and the levers that shorten it read fewer bytes per step or emit more tokens per step: 8-bit weights halve the weight traffic, and a draft whose proposals are accepted 70% of the time makes each target pass yield several tokens. The quality floor rules out the cheapest answer, 4-bit weights, which is the usual shape of a serving decision: the constraint that binds is often not a number on the dashboard but a quality measurement made elsewhere.

Real serving stacks combine all of these with prefix caching, chunked prefill and disaggregated pools (plates XXIII.6–8); the simulator here leaves those out to keep five levers.

Terms

TTFT
Time to first token: queueing plus prefill.
Time per output token
The interval between tokens during decoding.
Speculative decoding
Drafting tokens cheaply and verifying them in one target pass.
Quality floor
A limit on optimisations set by measured quality.
A serving design meets its targets by batching for throughput, then cutting bytes and passes per token, inside limits set by quality and cost.

Sources

  1. Yu et al., “Orca: A Distributed Serving System for Transformer-Based Generative Models”, OSDI 2022.
  2. Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention”, SOSP 2023.
  3. Leviathan, Kalman and Matias, “Fast Inference from Transformers via Speculative Decoding”, ICML 2023.
  4. Frantar et al., “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers”, ICLR 2023.