Drills: Kernels.
Five questions at a time from the whole volume, until a profile reads like a to-do list.
This plate draws rounds of five from the questions and predictions of the seven plates before it, with the reasoning and source plate after each answer.
The volume’s thread is traffic: most of what is won back inside a step is won by moving fewer bytes — fusing, tiling, never writing the scores, reading narrower weights — and the roofline says when that is the right fight.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Read a step profile.
- Decide when fusion and tiling help.
- Explain FlashAttention and the online softmax.
- Choose a low-precision format for a batch.
What to carry forward
A profile shows time, not FLOPs (plate XVI.1); fusion cuts the traffic of element-wise chains (plate XVI.2); the online softmax makes softmax a one-pass, block-wise computation (plate XVI.3), and FlashAttention uses it to keep attention on chip (plate XVI.4).
The best tile depends on the shape, so kernels are autotuned (plate XVI.5); narrow weights help memory-bound decoding and FP8 helps compute-bound work (plate XVI.6); and a cost model predicts it all well enough to argue with a profile (plate XVI.7). Part Four turns from training the model to shaping it: post-training.
Terms
- Fusion
- Several operations in one kernel.
- FlashAttention
- Tiled, exact attention that never stores scores.
- Autotuning
- Measuring candidate kernels and keeping the fastest.
- Weight-only quantisation
- Narrow weights, wide arithmetic.