Vol. III · Learning Machinery Plate 6 of 10LayerNorm, RMSNorm, normalisation, residual stream, training stabilityAL

Normalisation.

Whatever size a vector has grown to, rescale it before the next layer reads it.

As a vector passes through a deep network its size wanders: some layers enlarge it, some shrink it, and in a residual network it keeps accumulating additions. A layer that receives inputs ten times larger than usual produces outputs that may saturate, and a gradient step tuned for one scale is wrong for another. A normalisation layer takes each token’s vector and rescales it to a standard size, then multiplies by a learned gain (and, in LayerNorm, adds a learned bias), so the layer after it always sees inputs of a predictable size.

LayerNorm (2016) subtracts the vector’s mean and divides by its standard deviation. RMSNorm (2019) skips the mean and divides by the root mean square; it is cheaper and works as well, and most recent large models use it. In a Transformer the norm sits before each attention and MLP sub-layer — “pre-norm” — and once more before the output head (plate V.3).

Below: left, a 16-dimensional vector before and after LayerNorm and RMSNorm; shift and scale it with the sliders and watch the normalised versions stay put. Right, the size of the residual stream across 30 randomly initialised residual layers with no normalisation, with pre-norm, and with post-norm.

What you will learn

Compute LayerNorm and RMSNorm of a vector.
Explain what normalisation stabilises.
Compare LayerNorm with RMSNorm.
See how normalisation tames growth in a residual stream.

Workshop

Left: the input vector (grey), its LayerNorm (blue) and its RMSNorm (red), gains 1 and bias 0. Right: spread of the residual stream across 30 random residual layers (log scale) with no normalisation, with pre-norm (normalise each layer’s input) and with post-norm (normalise the stream itself).

A fixed size, whatever came before

The input here has mean 1.02 and spread 2.58; LayerNorm returns a vector with mean 0 and spread 1.000 whatever the shift and scale. RMSNorm divides only by the root mean square, so with a shift it keeps a positive mean (0.369); with no shift, the two agree closely. That is the case that matters in practice — Transformer activations hover around zero mean — which is why dropping the mean subtraction costs nothing and saves a reduction per vector.

The right panel shows what normalisation is for at depth. Thirty random residual layers that add their output to a running stream, with nothing to control it, grow the stream’s spread to about 4.1 × 10⁶. Normalising each layer’s input (pre-norm) leaves the additions bounded and the stream grows only slowly, to 4.73; normalising the stream itself after every addition (post-norm) pins it at 1.00. Each arrangement has consequences for training stability that plate V.3 measures. Normalisation layers carry learned gains, and some recent models add extra normalisation inside attention for stability (plate VII.7).

LayerNorm was introduced by Jimmy Ba, Jamie Kiros and Geoffrey Hinton in 2016 as a batch-independent alternative to batch normalisation, which normalises across examples and works poorly for sequences; RMSNorm was proposed by Biao Zhang and Rico Sennrich in 2019.

Terms

LayerNorm: Subtract the vector’s mean, divide by its standard deviation, then apply a learned gain and bias.
RMSNorm: Divide the vector by its root mean square, then apply a learned gain; no mean subtraction.
Pre-norm: Normalising the input of each sub-layer, leaving the residual stream itself unnormalised.
Gain: A learned per-dimension multiplier applied after normalising.

Normalisation rescales each token’s vector to a standard size before a layer reads it. RMSNorm, which skips the mean, is cheaper and now usual.