Vol. III · Learning Machinery Plate 8 of 10MLP, feed-forward network, GELU, ReLU, SwiGLU, nonlinearity, parametersA

The MLP.

Widen each token’s vector, bend it with a nonlinearity, squeeze it back: the part of the block that thinks about one token at a time.

After attention has mixed information between tokens, each Transformer block applies a multilayer perceptron to every token on its own: multiply the d-dimensional vector by a matrix that widens it to about 4d, apply a nonlinearity elementwise, and multiply by a second matrix back to d. Without the nonlinearity the two matrices would collapse into one linear map; with it, the MLP can compute features that are not linear in its input. Each column of the first matrix acts like a detector, and the nonlinearity decides how much each detector fires.

The nonlinearity has changed over time. ReLU, max(0, x), was simple and fast; GELU, a smooth version used by GPT-2 and BERT, weights each input by the probability that a standard normal falls below it; SiLU (also called swish) is x times the sigmoid of x. Gated variants such as SwiGLU multiply two projections together and are now the most common choice (plate VII.1). Two-thirds of the parameters of a standard Transformer block are in its MLP.

Below: left, the three nonlinearities. Right, inside micro: the share of MLP units that are switched on (GELU output above zero) at each of its four layers while it reads the prompt, and the parameter count of a block split between attention and MLP.

What you will learn

Describe the MLP as widen, nonlinearity, project back.
Compare ReLU, GELU and SiLU.
Count the parameters of an MLP and of attention in a block.
Measure how many units fire in a trained MLP.

Workshop

Left: ReLU, GELU and SiLU between −4 and 4. Right: the share of micro’s 512 MLP units per layer whose GELU output is positive, averaged over the prompt’s tokens; and the parameters of one micro block.

Detectors, then a choice

The three nonlinearities agree for large positive inputs and differ near zero: ReLU cuts sharply, GELU and SiLU bend smoothly and let small negative values through slightly — GELU(−1) is −0.159 — which gives a gradient everywhere and trains a little better. In micro, whose MLPs widen 128 dimensions to 512, the MLP holds 67% of each block’s parameters, as in any standard Transformer where the hidden width is four times d. Reading the prompt, about 22% of the first layer’s units have positive output and 21% of the last layer’s: each token switches on a different subset of detectors, a sparse pattern that interpretability methods (Vol. XXVIII) try to read.

MLPs are where much of a model’s stored knowledge appears to live: experiments that edit facts in a model usually change MLP weights in middle layers (plate VIII.8). They are also where a mixture of experts puts its extra capacity — many MLPs per layer, each token sent to a few (Vol. VII). Because an MLP processes every token independently, its cost is simple to count: 2 × 2 × d × 4d multiply-adds per token for the two matrices, the largest share of a dense model’s forward FLOPs (plate V.6).

The name “multilayer perceptron” goes back to Frank Rosenblatt’s perceptron of 1958, a single trainable linear threshold unit; two-layer networks with a nonlinearity between were shown in 1989 to be universal approximators, able to fit any continuous function given enough width.

Terms

MLP (feed-forward network): Two linear maps with a nonlinearity between, applied to each token separately.
Nonlinearity: An elementwise function such as ReLU, GELU or SiLU that lets the network compute non-linear features.
Hidden width: The width of the MLP’s middle layer, usually about four times d.
GELU: x times the standard normal cumulative distribution of x: a smooth ReLU.

The MLP widens, bends and projects back each token on its own, and holds about two-thirds of a standard block’s parameters.