Vol. XI · Optimisation at Scale Plate 4 of 10gradient clipping, gradient norm, threshold, safety, monitoring, instabilityLE

Gradient Clipping.

If the gradient is too long, shorten it — direction kept, size capped.

Global-norm clipping computes the length of the whole gradient, across every parameter, and if it exceeds a threshold c, multiplies the gradient by c divided by that length. The direction is unchanged; only the step size is capped. A threshold of 1.0 is the near-universal default, used by GPT-3 and Llama 2 among many others.

Clipping is insurance against a single unusual batch producing a gradient far larger than usual — exactly the situation in which an optimiser can take a damaging step. It is not free of side effects. When it triggers often, it is silently acting as a smaller learning rate; and a run that is clipped on every step may look stable while the underlying problem — a growing logit, a bad batch — goes unnoticed. The workshop reads the gradient norms recorded during training of micro, which was clipped at 1.0, and of the stability runs, which were not clipped at all.

Below: left, micro’s gradient norm over its 32 000 steps, with the clipping threshold. Right, the gradient norms of the 8-layer stability runs at three learning rates, without clipping.

What you will learn
Workshop
Left: micro’s gradient norm before clipping, sampled every 320 steps of its recorded run; the line is the chosen threshold. Right: largest gradient norm among the logged steps of the 8-layer stability runs (plate VII.7), which used no clipping, at learning rates 0.01, 0.03 and 0.1, log scale.

A safety net that changes the step

micro’s gradient norm starts near 0.64 and rises to about 1.61 by the end of training, a rise late in training that is commonly observed as the learning rate falls. With the threshold at 1.0, 57% of the logged steps were clipped, by a factor of 0.87 on average: for much of its run, micro’s effective learning rate was below its nominal one. With c = 2, 0% would have been.

The stability runs show what clipping guards against. Without it, the plain 8-layer model at learning rate 0.1 produced a gradient norm of 425, while the same model with QK-norm stayed 1,094 times lower. Clipping would have capped that step, but the cause — attention logits growing out of range — would have remained, and the run would still have ended badly (plate VII.7). That is the case for logging gradient norms (plate XV.6): a run that clips constantly is telling you something.

PaLM’s report notes that its loss spikes occurred despite gradient clipping being enabled. Clipping bounds a step’s size, not its direction, and a bounded step in a bad direction can still do damage.

Terms

Gradient norm
The length of the whole gradient vector across all parameters.
Global-norm clipping
Rescaling the gradient so that its norm does not exceed a threshold.
Threshold c
The largest gradient norm allowed; 1.0 is a common default.
Effective learning rate
The step actually taken, smaller than nominal when clipping triggers.
Clipping caps the step, not the cause: it guards against bad batches, and a run that clips constantly is hiding something worth seeing.

Sources

  1. Pascanu, Mikolov and Bengio, “On the difficulty of training recurrent neural networks”, ICML 2013 (gradient-norm clipping).
  2. Brown et al., NeurIPS 2020, Appendix B; Touvron et al., arXiv:2307.09288, 2023.