Residual Connections.
Instead of replacing the signal, each layer adds to it — and the gradient gets a highway back to the first layer.
In a plain deep network every layer replaces its input with something new, and the gradient that reaches the first layer is a product of one factor per layer. If those factors are a little below one — as they are for saturating activations like the sigmoid — the product vanishes; the early layers stop learning. In 2015, Kaiming He and colleagues made networks of more than a hundred layers trainable with a simple change: compute y = x + f(x) instead of y = f(x).
With the residual connection, the derivative of y with respect to x is the identity plus the derivative of f, so every layer passes the gradient back at least unchanged, plus a correction. The network becomes a running sum: an input, plus the contribution of layer 1, plus that of layer 2, and so on. The Transformer is built on this sum; it is called the residual stream (plate V.2).
Below: a deep network of random layers at initialisation. The curve shows the size of the gradient reaching each layer, from the last (right) back to the first (left), with and without residual connections.
- Write a residual connection and its derivative.
- See vanishing gradients in a deep plain network.
- Explain why residual connections keep gradients alive.
- Describe a residual network as a running sum of corrections.
A highway for the gradient
Through 30 sigmoid layers the gradient reaching the first layer of a plain stack is 2.3 × 10⁻¹⁹ times the one at the last layer — every sigmoid multiplies it by at most a quarter, and thirty quarters make an astronomically small number. Even ten layers cost a factor of 1.0 × 10⁻⁶. Add residual connections to the same random layers and the ratio is 0.10: a loss of about one order of magnitude instead of nineteen, because the identity path carries the gradient back past every sigmoid and each layer’s own derivative is only a correction on top. With tanh, which is steeper near zero, the plain stack loses less (0.60), but the principle is the same.
Residual connections also change what layers learn. Each layer starts close to doing nothing — f is small at initialisation — so a deep residual network begins life as a shallow one and adds depth only where it helps. In a Transformer, every attention layer and every MLP is wrapped this way: each reads the running sum, computes a correction and adds it back. Plate V.2 treats that sum as the model’s central data structure, and Vol. XXVIII reads it.
Terms
- Residual connection
- Adding a layer’s input to its output: y = x + f(x).
- Vanishing gradient
- A gradient shrinking towards zero as it passes back through many layers.
- Identity path
- The direct route through the additions, along which the gradient passes unchanged.
- Residual stream
- The running sum a Transformer carries from layer to layer (plate V.2).