Drills: The Transformer.
Five questions at a time from the whole volume, until a Transformer’s structure and arithmetic are second nature.
This plate draws rounds of five from the questions and predictions of the nine plates before it, with the reasoning and source plate after each answer.
Two kinds of knowledge from this volume are used everywhere later: the structure — embedding, blocks of norm–attention–add and norm–MLP–add, final norm, head — and the arithmetic — 12 L d² parameters, 2N and 6N FLOPs per token, 6ND for a run.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Recall the order of operations in a block.
- Count parameters and FLOPs from a configuration.
- Explain norm placement and weight tying.
- Tell the model families apart by their masks.
What to carry forward
A Transformer is an embedding, L identical blocks — x + Attn(Norm x), then x + MLP(Norm x) — a final norm and an output head (plate V.1). The blocks communicate through the residual stream, which can be read at any depth (plate V.2); putting the norm before each sub-layer keeps that stream’s gradient path clean (plate V.3); and the decoder-only family, causal everywhere, is the one this course follows (plate V.4).
The arithmetic: about 12 d² parameters per block plus V d for the vocabulary (plate V.5), 2N FLOPs per forward token and 6N per training token (plate V.6), a head that can share the embedding table (plate V.7). A real one runs in the page (plate V.8) and trains in it, slowly (plate V.9). Vol. VI turns to what this volume took for granted: how a model knows where each token is.
Terms
- Block
- Norm, attention, add; norm, MLP, add.
- 12 L d²
- Non-embedding parameters of a standard model.
- 6ND
- Training FLOPs of a run.
- Weight tying
- Embedding table reused as output head.