Vol. XXIII · Serving Plate 7 of 9disaggregated serving, prefill, decode, DistServe, Splitwise, KV transfer, DeepSeek-V3, tail latencyCE

Splitting Prefill From Decode.

Give the compute-bound phase and the memory-bound phase machines of their own.

Prefill and decode want different things (plate XXIII.1): prefill is a burst of arithmetic on one request, decode a long stream of memory-bound steps for many. Mixed on one GPU, they interfere — each admitted prompt’s prefill delays the next decode step for every running user, so the time per token jumps whenever a long prompt arrives (plate XXIII.4).

Disaggregated serving — DistServe (Zhong et al., 2024) and Splitwise (Patel et al., 2024) — puts prefill on one pool of GPUs and decode on another: a request is prefilled, its KV cache is sent over the network to a decode GPU, and decoding proceeds there without interruption. DeepSeek serves V3 this way, with separate prefill and decode deployments of their own sizes. The workshop compares two H100s serving Llama 3 8B as two independent replicas with the same two GPUs split into one prefill and one decode GPU, with caches moved at 50 GB/s.

Below: left, 95th-percentile time per output token against the total request rate for both arrangements; right, the 95th-percentile time to first token.

What you will learn
Workshop
Simulated: Llama 3 8B in BF16, two H100s in total, Poisson arrivals, prompts exponential with mean 2 048 tokens, outputs mean 256. Colocated: two replicas, each a GPU doing both phases with continuous batching and half the traffic. Disaggregated: one GPU prefills, one decodes; each cache crosses a 50 GB/s link. Times are roofline estimates from the H100’s published specifications (989 TFLOP/s dense BF16, 3.35 TB/s), assuming 60% of peak compute and 80% of peak bandwidth are reached plus 0.2 ms per pass — not measurements.

Steady tokens, a second queue

At 12 requests a second, the colocated replicas’ 95th-percentile time per token is 17.9 ms and the disaggregated pair’s 9.5 ms: with no prefills interrupting it, the decode GPU’s steps are uniform. Moving a 2 048-token cache across the 50 GB/s link takes about 5.4 ms, small beside a prefill. But the prefill GPU now has its own queue, and at 20 requests a second it is the bottleneck: the 95th-percentile first-token wait is 1.43 s disaggregated against 0.34 s colocated, while time per token stays at 13.2 ms against 30.9 ms.

So disaggregation is a trade: steadier decoding and the freedom to size, parallelise and even choose hardware for each phase separately, against cache transfer and the need to balance the two pools to the traffic’s mix of prompt and output lengths. It pays most for large models with long prompts and strict per-token targets. DeepSeek’s V3 deployment shows the scale: prefill units of 32 GPUs and decode units of 320, each with its own parallelism. Chunked prefill is the colocated alternative, bounding each step’s prefill work instead of moving it elsewhere.

The cache transfer is why disaggregated deployments care about fast interconnects, and why compact caches — grouped-query or latent attention, quantised caches — make disaggregation cheaper.

Terms

Disaggregated serving
Running prefill and decode on separate pools of GPUs.
Interference
One phase’s work delaying the other’s on shared hardware.
KV transfer
Sending a prefilled request’s cache to the decode pool.
Tail latency
The slow end of the latency distribution, such as the 95th percentile.
Disaggregation removes prefill stalls from decoding, at the cost of moving caches and balancing two pools.

Sources

  1. Zhong et al., “DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving”, OSDI 2024.
  2. Patel et al., “Splitwise: Efficient Generative LLM Inference Using Phase Splitting”, ISCA 2024.
  3. DeepSeek-AI, “DeepSeek-V3 Technical Report”, arXiv:2412.19437, 2024, section 3.4.