Drills: Learning Machinery.
Five questions at a time from the whole volume, until the parts of a trainable network are familiar.
This plate draws rounds of five from the questions and predictions of the nine plates before it, with the reasoning and the source plate shown after each answer.
The volume’s parts reappear in every model that follows: embeddings and a softmax at the two ends, MLPs, residual connections and norms in between, backpropagation and AdamW to train them, initialisation to let training start, and held-out loss to say whether it worked.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Recall the formulas of softmax, cross-entropy and Adam.
- Reason about signal and gradient size through depth.
- Count MLP and embedding parameters.
- Read overfitting from curves.
What to carry forward
A model starts from embeddings and ends in a softmax whose cross-entropy gradient, p − y, is the only signal from the data (plates III.1–III.2). Backpropagation carries that signal to every parameter for about twice the cost of the forward pass, and AdamW turns gradients into steps that are robust to scale (plates III.3–III.4).
Depth is made trainable by three devices: initialisation that keeps signals the same size, normalisation that rescales each layer’s input, and residual connections that give gradients a direct path (plates III.5–III.7). MLPs hold most parameters (plate III.8), and held-out loss, not training loss, says whether any of it worked (plate III.9). Vol. IV adds the one missing piece — attention.
Terms
- Cross-entropy gradient
- p − y on the logits.
- AdamW
- Adam with weight decay applied separately.
- He initialisation
- Variance 2/n for ReLU layers.
- Residual connection
- y = x + f(x).