Vol. XIII · Hardware Plate 4 of 9arithmetic intensity, batch size, matrix multiply, prefill, decode, ridge point, memory-boundCE

Arithmetic Intensity in a Transformer.

Read the weights once, use them for every token in the batch — so the batch size is the intensity.

Multiplying m token vectors of width d by a d × d weight matrix costs 2 m d² FLOPs and moves about 2 d² bytes of weights in BF16, plus the activations. For m much smaller than d, the intensity is roughly m FLOPs per byte: each token in the batch reuses every weight once. On a chip whose ridge point is about 300, the multiply becomes compute-bound only when about 300 tokens share each weight read.

This simple fact organises much of how models are run. Training and prefill — reading a prompt — process thousands of tokens per weight read and sit comfortably above the ridge. Decode — generating one token at a time for one user — has m = 1 and is deeply memory-bound, so serving systems batch many users’ decode steps together to raise m (plate XXIII.4). The workshop computes the intensity and time of one projection matrix as the batch grows.

Below: left, arithmetic intensity of a d × d multiply against the number of tokens, with the ridge points of three accelerators. Right, time per token on the selected accelerator — flat while memory-bound costs are shared, then constant once compute-bound.

What you will learn

Derive a matrix multiply’s intensity as a function of tokens.
Find the batch size at which it becomes compute-bound.
Contrast prefill with decode.
Explain why serving batches requests.

Workshop

One d × d projection in BF16, m tokens: FLOPs 2md², bytes 2(d² + 2md). Left: intensity against m with the BF16 ridge points of A100, H100 and B200. Right: time per token on the selected accelerator, from the roofline.

The batch size is the intensity

For a 4 096-wide projection on an H100, intensity rises almost exactly with the number of tokens until about 512 tokens, where it crosses the ridge point of 295. Below that, the time is set by reading the 32 MB of weights, and one token costs 10.02 µs while a batch shares the same read; above it, each token costs only its own arithmetic, 0.0339 µs, 295 times less. On an A100 the crossing comes at about 256 tokens: the H100’s arithmetic grew about threefold over the A100’s but its bandwidth only about 1.6-fold, so it needs larger batches to be used fully. The B200 raised both together, and crosses at about 512, as the H100 does.

This is why the same model is cheap to train and expensive to serve one user at a time. Training batches millions of tokens; a prompt of a few thousand tokens is processed at full speed; but each generated token for one user reads every weight for one multiply. Serving systems therefore pack many users’ decode steps into one batch (plate XXIII.4), and techniques such as speculative decoding (plate XXIV.4) exist to give the weights more work per read.

Attention during decode is worse still: each user’s key–value cache is read for that user alone, so batching does not raise its intensity, and long contexts make it the dominant cost of generation (plate XXIII.2).

Terms

Prefill: Processing a prompt’s tokens in parallel; high intensity.
Decode: Generating tokens one at a time; low intensity.
Weight reuse: Using each weight read from memory for many tokens.
Batching: Processing several sequences’ tokens with one read of the weights.

A weight read once serves every token in the batch, so intensity ≈ tokens per read; below the ridge point — a few hundred tokens — work is memory-bound.

Sources

NVIDIA A100, H100 and HGX B200 datasheets.
Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention”, SOSP 2023 (batching in serving).