Vol. XXIII · Serving Plate 1 of 9prefill, decode, roofline, memory bandwidth, arithmetic intensity, time to first token, inferenceCE

Prefill and Decode.

The prompt is read all at once. The answer is written one token at a time, and each token costs a full read of the weights.

A served request runs the model in two phases. Prefill processes the prompt: all its tokens go through the network in one forward pass, filling the key–value cache (plate XXIII.2) and producing the first output token. Decode then generates the rest, one token per forward pass, because each token depends on the one before.

The two phases hit different limits of the hardware. A forward pass must read every weight from memory once, however many tokens it processes; the arithmetic grows with the number of tokens. With few tokens per pass the GPU waits on memory, with many it waits on arithmetic — the roofline of plate XIII.3. The workshop computes the time per token of one forward pass of Llama 3 8B on an H100 as the number of tokens processed together grows. Times are roofline estimates from the H100’s published specifications (989 TFLOP/s dense BF16, 3.35 TB/s), assuming 60% of peak compute and 80% of peak bandwidth are reached plus 0.2 ms per pass — not measurements.

Below: time per token of one forward pass against the number of tokens it processes, on a logarithmic scale; one token is a decode step for one user, thousands are a prefill.

What you will learn
Workshop
Llama 3 8B (8.03 billion parameters, 16.1 GB in BF16) on one H100. One forward pass processing the chosen number of prompt tokens, including attention among them. Decode: one new token reading a 1 000-token cache.

Arithmetic for the prompt, memory for the answer

Prefilling a 1 024-token prompt takes about 28.4 ms — 0.0277 ms per token — because each read of the 16.1 GB of weights feeds a thousand tokens’ worth of arithmetic. Generating one token for one user takes 6.24 ms: the same full read of the weights for a single token, 225 times as long per token. One user is served about 160 tokens per second however fast the GPU’s arithmetic is.

The arithmetic intensity of a pass is roughly the number of tokens in it, in FLOPs per byte of weights read; the H100’s ridge is about 295, so a pass needs a few hundred tokens before arithmetic, not memory, sets its time. Prefill gets there from one long prompt. Decode gets there only by batching many users’ next tokens into one pass (plate XXIII.4), which is the central problem of serving. The two phases also map onto the two latencies users feel: prefill sets the time to the first token, decode the time between tokens.

Long prompts change the picture slightly: attention among prompt tokens grows with the square of the prompt length, so the time per token of prefill creeps up past a few thousand tokens.

Terms

Prefill
Processing the whole prompt in one forward pass.
Decode
Generating output one token per forward pass.
Memory-bound
Limited by how fast weights can be read, not by arithmetic.
Arithmetic intensity
FLOPs per byte read: roughly the tokens per pass here.
Prefill is limited by arithmetic, decode by reading the weights once per token; batching is how decode gets its arithmetic back.

Sources

  1. Pope et al., “Efficiently Scaling Transformer Inference”, MLSys 2023.
  2. NVIDIA H100 Tensor Core GPU datasheet (SXM5).
  3. Llama Team, AI @ Meta, “The Llama 3 Herd of Models”, arXiv:2407.21783, 2024, Table 3.