Vol. III · Learning Machinery Plate 3 of 10backpropagation, chain rule, computation graph, autograd, gradientL

Backpropagation.

Record every operation on the way forward; walk the record backwards multiplying local derivatives; every parameter gets its gradient at once.

A language model has millions or billions of parameters and one loss. To improve it, each parameter needs its gradient: how much the loss would change if that parameter moved a little. Nudging each parameter in turn and re-running the model would take one forward pass per parameter. Backpropagation gets all of them in one backward pass, by the chain rule: the derivative of the loss with respect to a value is the sum, over everything that value fed into, of the derivative with respect to that thing times the local derivative of the operation.

Software makes this automatic. As the forward pass runs, every operation is recorded on a tape together with what it needs to compute its own local derivatives; the backward pass then walks the tape in reverse. The course’s own engine (js/tensor.js) does exactly this for every operation a GPT needs, and is checked against PyTorch’s autograd on the same weights. Below, the same mechanism on a single neuron small enough to see whole.

Below: one neuron, y = tanh(w₁x₁ + w₂x₂ + b), drawn as a graph. Each box shows its value on the way forward and its gradient on the way back. The readings compare the backpropagated gradient of w₁ with one estimated by nudging w₁ and re-running.

What you will learn

Draw a computation as a graph of simple operations.
Apply the chain rule backwards through the graph.
Check a gradient against a finite difference.
Explain why backpropagation costs about as much as the forward pass.

Workshop

The neuron as a computation graph. In each box: the value computed on the way forward, and below it the gradient of y with respect to that value, computed on the way back.

The chain rule, run backwards

At the default settings the output is y = −0.1974, and walking the graph backwards gives ∂y/∂s = 1 − tanh²(s) = 0.9610, then ∂y/∂w₁ = 1.92209, which is ∂y/∂s times x₁. Nudging w₁ up and down by 0.0001 and running the neuron twice estimates the same derivative as 1.92209; the two differ by 2.3 × 10⁻⁸, rounding error. Push the inputs and weights to 2 and the sum is so large that tanh saturates: ∂y/∂s falls to 8.2 × 10⁻⁹ and almost no gradient reaches any weight — a vanishing gradient, the reason for much of the machinery in plates III.5 to III.7.

In a real network the same walk happens over a graph of millions of operations, and it is cheap: each operation’s backward step costs about as much as its forward step (a matrix product’s backward is two matrix products, so the whole backward pass costs about twice the forward). That is where the factor of three in “6N FLOPs per training token” comes from: 2N forward plus 4N backward (plate V.6). The price is memory: everything the backward pass needs from the forward pass — the activations — must be kept until it is used, which is why activation memory and recomputation get a plate of their own (XIV.2).

Backpropagation was derived several times — by Seppo Linnainmaa as reverse-mode automatic differentiation in 1970, by Paul Werbos for neural networks in 1974 — and made famous for neural networks by Rumelhart, Hinton and Williams in 1986.

Terms

Gradient: The vector of partial derivatives of the loss with respect to every parameter.
Chain rule: The derivative of a composition is the product of the derivatives of its parts.
Backpropagation: Computing all gradients in one backward pass through the recorded computation.
Finite difference: Estimating a derivative by nudging an input and re-running; used to check gradients.

Backpropagation is the chain rule applied backwards through a recorded computation. One backward pass, about twice the cost of the forward, gives every gradient.