Vol. XVI · Kernels and Efficiency Plate 7 of 8cost model, roofline, prediction, FLOPs, bytes, launch overhead, achievable efficiency, Transformer layerCE

Predicting a Kernel’s Time.

FLOPs over achievable peak, bytes over achievable bandwidth, the larger of the two, plus a launch — the estimate behind every optimisation decision.

The roofline (plate XIII.3) gives a lower bound on a kernel’s time. A working estimate adds three corrections: real kernels reach only a fraction of peak arithmetic — 60–80% for a well-tuned large multiply — and of peak bandwidth — around 80–90% for streaming kernels; every launch costs a few microseconds; and the two limits do not perfectly overlap. The estimate is t ≈ max(F / (e_c · peak), B / (e_m · bandwidth)) + t_launch.

Such a model is how engineers decide what to optimise before profiling, and how they check a profile once they have one: a kernel far slower than its estimate has a problem worth finding. The workshop applies it to every operation of one Llama 3 8B layer on an H100, for a chosen number of tokens, and compares the total with a FLOPs-only estimate and a bytes-only one.

Below: the predicted time of each operation in one layer — red if memory-bound, blue if compute-bound — and readouts comparing the total with simpler estimates.

What you will learn
Workshop
One Llama 3 8B layer’s forward pass on a single sequence of the chosen length (attention unfused, as in blockops), on an H100. Each operation’s FLOPs and bytes from plate XIII.4; achievable fractions and launch overhead as chosen. FLOPs-only: ΣF ÷ (e_c · peak). Bytes-only: ΣB ÷ (e_m · bandwidth).

An estimate to argue with

For one 8 192-token sequence, the model predicts 8,756 µs for the layer — 1.47 times a FLOPs-only estimate of 5,965 µs, because 8 of the operations are memory-bound and take 41% of the time; the materialised attention scores and softmax dominate them. At 128 tokens the picture changes: the layer takes 205 µs, 2.53 times the FLOPs-only figure, because every operation is small enough for bytes and launches to rule — 18% of the time is launch overhead alone. At 32 768 tokens the unfused attention swells to dominate everything: 71,117 µs.

The model is wrong in detail and right in direction, which is what it is for. It says what to fix first (fuse the attention, plate XVI.4; fuse the element-wise chains, plate XVI.2), what not to bother with (a large multiply already near its achievable peak), and what a profile should show. When the profile disagrees — a kernel at half its predicted speed — the gap itself is the finding: a poor tile choice (plate XVI.5), an uncoalesced memory access, or an unexpected synchronisation.

The achievable fractions are themselves measured, kernel by kernel, by tools such as Nsight Compute, which report a kernel’s achieved FLOP/s and bandwidth as a percentage of peak — the roofline drawn from real data.

Terms

Cost model
A formula predicting a kernel’s time from its FLOPs and bytes.
Achievable peak
The fraction of the specification’s peak a good kernel actually reaches.
Launch overhead
The fixed time to start a kernel.
Speed of light
Engineers’ name for the roofline bound a kernel cannot beat.
Predict a kernel as max(FLOPs ÷ achievable peak, bytes ÷ achievable bandwidth) + launch; summed over a layer it says what to optimise and what a profile should show.

Sources

  1. Williams, Waterman and Patterson, “Roofline: An Insightful Visual Performance Model”, CACM 52(4), 2009.
  2. NVIDIA Nsight Compute documentation (achieved FLOP/s and bandwidth per kernel).