Vol. III · Learning Machinery Plate 5 of 10initialisation, Xavier, He, variance, deep networks, signal propagationAL

Initialisation.

Draw the random starting weights a little too small and the signal vanishes after twenty layers; a little too large and it explodes.

A layer multiplies its input by a weight matrix. If the weights are random with standard deviation σ and the layer has n inputs, each output is a sum of n random terms, and its spread is about σ√n times the spread of the input. Stack thirty such layers and that factor is raised to the thirtieth power: anything but exactly the right scale either shrinks the signal towards zero or blows it up. Gradients flowing backwards suffer the same fate.

The cure is to choose σ so that the factor is one. For a linear or tanh layer that is σ = 1/√n (Xavier or Glorot initialisation, 2010); a ReLU zeroes half its inputs, so it needs √2 times more, σ = √2/√n (He initialisation, 2015). Transformers add their own rules on top — small initial weights, and output projections scaled down by the square root of the number of layers so that the residual stream does not grow with depth (Vol. V).

Below: a vector passed through 30 random layers of width 128 at initialisation. The curve shows the spread (standard deviation) of the signal after each layer, on a log scale, for the scale and nonlinearity chosen; the dashed line is He initialisation with ReLU.

What you will learn

Explain why the scale of random weights compounds with depth.
Derive the 1/√n rule for keeping variance constant.
Explain why ReLU needs √2 times larger weights.
See vanishing and exploding signals directly.

Workshop

Standard deviation of the activations after each of 30 random layers (log scale), for weights drawn with standard deviation scale/√128. Dashed: ReLU with He initialisation (scale √2).

One number, raised to the thirtieth power

With ReLU and weights at scale 1/√n — right for a linear layer, wrong for ReLU — the signal shrinks by about a factor of √2 per layer and after 30 layers its spread is 1.4 × 10⁻⁵: the deep layers receive essentially nothing. At scale 2 it grows instead, to 1.5 × 10⁴. At He’s scale √2 it stays the same order of magnitude — 0.45 after 30 layers, against 0.92 at the start, the difference being the randomness of a finite width. Remove the nonlinearity and scale 1 is the right one (0.53); with tanh at scale 1 the spread settles near 0.12, because tanh squashes large values and is nearly linear for small ones.

Modern language models are not plain stacks like this — residual connections (plate III.7) and normalisation (plate III.6) change the arithmetic — but the principle survives. They initialise weights from a small normal distribution (GPT-2 used standard deviation 0.02), scale the projections that write into the residual stream by 1/√(2L) so that adding 2L of them does not blow it up, and, in schemes such as μP (plate XI.8), set every layer’s scale by width so that the right learning rate does not change as the model grows. A bad initialisation rarely shows as an error message; it shows as a loss that refuses to fall, or a loss spike in the first few hundred steps.

Xavier Glorot and Yoshua Bengio’s 2010 paper diagnosed why deep networks were so hard to train — saturated activations and badly scaled initial weights — and Kaiming He and colleagues’ 2015 paper derived the ReLU correction while training the first networks that beat human-level accuracy on ImageNet.

Terms

Initialisation: The random starting values of a network’s weights.
Xavier (Glorot) initialisation: Weights with variance 1/n, keeping a linear or tanh layer’s output variance equal to its input’s.
He initialisation: Weights with variance 2/n, compensating for ReLU zeroing half its inputs.
Vanishing and exploding signals: Activations or gradients shrinking to zero or growing without bound through many layers.

Each layer multiplies the signal’s spread by about σ√n; depth raises that to a power. Initialise so the factor is one — 1/√n, or √2/√n for ReLU — and signals survive depth.