Drills: Architectures.
Five questions at a time from the whole volume, until the parts of a 2025 model are as familiar as those of a 2017 one.
This plate draws rounds of five from the questions and predictions of the nine plates before it, with the reasoning and source plate after each answer.
Two kinds of change run through the volume: changes that make a model better for the same cost — gates, balanced experts, stable logits, sensible shapes — and changes that make it cheaper for nearly the same quality — shared or compressed keys and values, sparse experts, attention without the square.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Recognise the components of current open models.
- Separate total from active parameters.
- Compute key–value cache savings.
- Name the usual fixes for training instability.
What to carry forward
A 2025 open model differs from the 2017 Transformer in a handful of parts: a gated MLP (plate VII.1), keys and values shared across heads or compressed into a latent (plates VII.2–VII.3), pre-norm with RMSNorm and rotary positions (Vols. V–VI), and, increasingly, experts routed per token (plates VII.4–VII.6). Each was measured on the course corpus: SwiGLU gained a little at equal parameters, GQA cost little, and the MoE bought a clear gain at equal compute.
Around those parts sit the measures that keep training on the rails — QK-norm, z-loss and soft-capping (plate VII.7) — and the alternatives that trade exact recall for constant memory (plate VII.8). And comparing shapes means checking what a parameter count leaves out (plate VII.9). Part Two now turns from the model to its training: the objective, the data, and the scaling laws that decide how big to build.
Terms
- SwiGLU
- A gated MLP with SiLU.
- GQA
- Several query heads sharing one key–value head.
- MoE
- Many experts, a few active per token.
- QK-norm
- Normalised queries and keys.