Vol. XXIV · Making Inference Cheap Plate 7 of 10pruning, sparsity, magnitude pruning, Wanda, SparseGPT, 2:4 sparsity, structured sparsityAE

Pruning and Sparsity.

Set half the weights to zero. Which half decides whether the model notices.

Many weights of a trained network are small, and removing them changes little. Magnitude pruning zeros the smallest; Wanda (Sun et al., 2024) scores each weight by |w| times the norm of its input across calibration text, because a small weight on a large input matters more than a large weight on an input that is always near zero — the same insight as AWQ’s (plate XXIV.2). SparseGPT (Frantar and Alistarh, 2023) adds GPTQ-style error compensation and prunes models with over a hundred billion parameters to half density with small loss.

Zeros save memory and time only if they are stored and computed sparsely. Unstructured sparsity — zeros anywhere — is hard for GPUs to exploit; NVIDIA’s tensor cores since the A100 support a 2:4 pattern, at most two non-zeros in every four consecutive weights, and can skip the zeros. The course model micro (1.8 million parameters, four layers; plate I.9) is quantised, pruned and sped up in the page, and its loss is measured on held-out course text it never trained on. The workshop prunes every linear layer of micro, per output column, by magnitude or by Wanda (calibrated on English course text), or to the 2:4 pattern by Wanda score.

Below: held-out loss increase against the share of weights removed, for magnitude and Wanda pruning, with 2:4 at one half marked; readouts for the chosen setting.

What you will learn
Workshop
micro, every attention and feed-forward matrix; for each output column, the inputs with the lowest scores set to zero. Wanda input norms from about 500 tokens of English held-out course text. Loss on the first 1 500 characters of the English and 500 of the Chinese held-out plates; no retraining after pruning.

Which half, and in what pattern

Removing 20% of micro’s weights costs 0.103 nats per token by magnitude and 0.050 by Wanda. At half, the gap opens: 0.896 against 0.614; at 80% the model is badly damaged either way (3.32 and 2.88). Forcing the 2:4 pattern at the same half costs 1.157 — more than unstructured Wanda, because within each group of four the two zeros must fall there even when the group’s weights all matter.

That cost buys speed: 2:4 weights can be stored in a little over half the space and multiplied by tensor cores that skip the zeros, while unstructured zeros at one half save little on a GPU. Pruning has been less used for language-model serving than quantisation, because quantisation gives larger, hardware-friendly savings at smaller cost and the two do not combine as easily; structured removal of whole heads, layers or channels, followed by retraining, is the form that has made production models — NVIDIA’s Minitron models were pruned from larger ones and then distilled.

Optimal Brain Damage (1989) pruned by second-order importance; SparseGPT and GPTQ descend from the same line of work, which is why pruning and quantisation share their mathematics.

Terms

Pruning
Setting weights to zero.
Wanda
Pruning by weight magnitude times input norm.
Unstructured sparsity
Zeros anywhere, hard to accelerate.
2:4 sparsity
At most two non-zeros in every four weights, accelerated by tensor cores.
Pruning by weight times input beats pruning by weight alone; zeros pay off only in patterns hardware can skip.

Sources

  1. Sun, Liu, Bair and Kolter, “A Simple and Effective Pruning Approach for Large Language Models” (Wanda), ICLR 2024.
  2. Frantar and Alistarh, “SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot”, ICML 2023.
  3. Mishra et al., “Accelerating Sparse Deep Neural Networks”, arXiv:2104.08378, 2021 (2:4 sparsity on NVIDIA Ampere).
  4. LeCun, Denker and Solla, “Optimal Brain Damage”, NeurIPS 1989.