Vol. XIII · Hardware Plate 3 of 9roofline model, arithmetic intensity, memory-bound, compute-bound, ridge point, performance modelCE

The Roofline.

Plot each operation by how many FLOPs it does per byte it moves; the chip’s two limits form a roof it cannot rise above.

Every operation does some arithmetic and moves some bytes to and from memory. If it can run at the chip’s peak FLOP/s, its time is FLOPs ÷ peak; if memory cannot feed it fast enough, its time is bytes ÷ bandwidth. The larger of the two wins. Dividing FLOPs by bytes gives the arithmetic intensity I, and the attainable speed is min(peak, bandwidth × I): a sloped roof for low intensity, a flat one for high, meeting at the ridge point peak ÷ bandwidth — about 295 FLOPs per byte for an H100 in BF16.

The roofline model (Williams, Waterman and Patterson, 2009) turns performance from a mystery into arithmetic. The workshop places every operation of one Llama 3 8B-sized Transformer block on an H100’s roofline, for the chosen number of tokens processed together: the large matrix multiplies sit on the flat roof, while norms, residual additions, softmax and the attention-score operations lie on the slope, however fast the chip.

Below: left, the roofline of the selected accelerator and the operations of one block at the chosen batch of tokens. Right, the time each operation takes and whether it is compute- or memory-bound.

What you will learn
Workshop
One Transformer block of the published Llama 3 8B configuration (d = 4 096, 32 heads, 8 key–value heads, SwiGLU width 14 336), forward pass over the chosen number of tokens in sequences of 8 192, BF16. FLOPs and bytes per operation from the course’s cost model (weights read once; activations read and written once; attention scores materialised). Left: attainable FLOP/s against intensity, log–log. Right: time per operation.

Two roofs and a ridge

On an H100 the ridge point is 295 FLOPs per byte. With 8 192 tokens processed together, the projections and MLP matrices have intensities in the thousands and sit on the flat roof; 8 of the 12 operations — the norms, residual additions, activation, softmax and the two attention-score operations, whose score matrices are written to and read from HBM — lie on the slope. They perform only 13.5% of the block’s FLOPs but take 46% of its time.

With a single token — generation for one user — every operation falls below the ridge, because each weight is read to multiply just one vector: the whole block is memory-bound, which is why generating text is limited by bandwidth (plate XXIII.1). With 65 536 tokens the matrix multiplies saturate the roof and the attention scores dominate. The remedies the following volumes describe all move operations up and to the right: batching more tokens per weight read, fusing element-wise operations into the matrix multiplies (plate XVI.2), and never materialising the attention scores (plate XVI.4).

The cost model here counts each operand read once from HBM. Real kernels may read some data several times or keep some in cache, so measured intensities differ; profilers report achieved bytes and FLOPs directly (plate XVI.1).

Terms

Arithmetic intensity
FLOPs performed per byte moved to or from memory.
Ridge point
The intensity at which an operation stops being memory-bound: peak ÷ bandwidth.
Memory-bound
Limited by bandwidth; more FLOP/s would not help.
Compute-bound
Limited by arithmetic throughput; more bandwidth would not help.
An operation’s arithmetic intensity against the ridge point decides whether bandwidth or arithmetic limits it — and memory-bound work takes time out of all proportion to its FLOPs.

Sources

  1. Williams, Waterman and Patterson, “Roofline: An Insightful Visual Performance Model for Multicore Architectures”, Communications of the ACM 52(4), 2009.
  2. NVIDIA H100 datasheet; Llama 3 8B configuration (arXiv:2407.21783).