Vol. V · The Transformer Plate 1 of 10Transformer block, attention, MLP, LayerNorm, residual connection, pre-normA

The Transformer Block.

Norm, attend, add; norm, think, add. Repeat L times. That is the whole architecture.

Every part of the previous two volumes appears in one block. The token vectors arrive as a sequence x. The block normalises them, lets each position attend to the earlier ones and adds the result back to x; then normalises again, runs the MLP on each position separately and adds that back too. Attention moves information between positions; the MLP processes each position on its own; norms keep the sizes steady; the residual additions keep the gradient’s highway open.

A Transformer language model is an embedding table, a stack of L such blocks with their own weights, a final norm and an output head. Nothing else. micro has four blocks; GPT-3 had 96. The workshop follows the vectors for the last token of a prompt through micro’s four blocks and shows how large each block’s two contributions are compared with the stream they are added to.

Below: left, one block as a diagram. Right, inside micro reading the prompt: for the last token, the size of the running vector entering each block, and the sizes of what that block’s attention and MLP add to it.

What you will learn

Name the parts of a pre-norm Transformer block in order.
Say what attention does and what the MLP does in a block.
Assemble a full model from blocks, an embedding and a head.
Compare the sizes of the updates each sub-layer makes.

Workshop

Left: a pre-norm Transformer block. Right: for the prompt’s last token in micro, the length of the residual vector entering each block (grey) and of the attention (blue) and MLP (red) outputs added in that block.

Two sub-layers, two additions

For the last token of the prompt, the vector entering micro’s first block has length 0.61 — the embedding plus its position, both small at the start. Block 1’s attention and MLP add updates whose combined length is 10.9 times that: the first block mostly writes the stream rather than editing it. By the fourth block the stream has grown to length 7.80, and that block’s two updates together are 1.41 times it — still large, because micro has only four blocks to do all its work. The vector leaves the stack 20 times longer than it entered. In models with dozens of layers the stream accumulates many contributions, each a smaller share of the whole, and that is what plate V.2 exploits to read the sum.

The design has been remarkably stable. The 2017 original placed the norm after each addition (post-norm, plate V.3), used ReLU in the MLP and sinusoidal positions; today’s models use pre-norm with RMSNorm, gated MLPs and rotary positions (Vol. VII), but the skeleton — attention then MLP, each wrapped in norm and residual — is unchanged. Because every block has the same shape, scaling a model up is mostly a matter of choosing width d, depth L and head count, which plate V.5 turns into a parameter count.

The Transformer was introduced in “Attention Is All You Need” by Ashish Vaswani and seven co-authors at Google in June 2017, for translation, with an encoder and a decoder; the decoder half alone, trained on next-token prediction, became GPT a year later.

Terms

Transformer block: Norm, attention, residual add; norm, MLP, residual add.
Sub-layer: One of the two halves of a block: attention or MLP.
Stack: L identical blocks with separate weights, applied in sequence.
Output head: The final norm and matrix that turn the last vectors into logits.

A block is x + Attn(Norm x), then x + MLP(Norm x). A Transformer is an embedding, L blocks and an output head.