Smarter Quantisation.
What matters is not the error in the weights but the error in the outputs, on the inputs the layer actually sees.
A linear layer’s output error is x·(Ŵ − W): weight errors matter in proportion to the inputs they multiply. Two widely used methods exploit this with a little calibration text. GPTQ (Frantar et al., 2023) quantises the weights for one input dimension at a time and adjusts the not-yet-quantised weights to cancel the error, weighted by the inverse of the inputs’ second-moment matrix H = XᵀX — an approximation to second-order pruning methods from the 1990s. AWQ (Lin et al., 2024) observes that input channels with large average activations matter most, scales those weights up before rounding (and the inputs down to compensate), and picks the scaling strength by a small grid search on the calibration output error.
The course model micro (1.8 million parameters, four layers; plate I.9) is quantised, pruned and sped up in the page, and its loss is measured on held-out course text it never trained on. The workshop quantises every linear layer of micro with one scale per column by round-to-nearest, AWQ or GPTQ, calibrating on English course text, and measures the loss on English and on Chinese held-out text.
Below: loss increase over 8-bit micro for each method at the chosen bit width, on English and on Chinese held-out text; calibration used English only.
- Explain output-error-aware quantisation.
- Describe GPTQ’s error compensation.
- Describe AWQ’s activation-aware scaling.
- See calibration data shape the result.
Fit the inputs, not the weights — but which inputs?
At 3 bits, on English text like its calibration data, GPTQ cuts the loss increase from 1.508 (round to nearest) to 1.301, and AWQ to 1.348. On Chinese the picture changes: AWQ roughly matches plain rounding (1.407 against 1.401), and GPTQ is much worse, at 1.778. GPTQ has fitted each layer’s weights to the statistics of English inputs, moving error into directions English rarely uses and Chinese does; AWQ changes only one scale per input channel and so has far less to overfit. At 4 bits all three are closer (0.339, 0.296 and 0.324 on English).
The AWQ authors made this argument about calibration robustness, and the lesson generalises: a calibration set is training data for the quantiser, and it should cover what the model will be used for — languages, code, long contexts. Both methods are fast enough to run on a single GPU for a model with tens of billions of parameters and are standard in open-model deployment, usually at 4 bits with groups of 128; below 4 bits, quantisation-aware training, which lets the model adapt to its rounding, does better than any post-training method.
Terms
- Calibration data
- Sample inputs used to fit a quantiser.
- GPTQ
- Sequential quantisation that compensates each rounding error using the inputs’ inverse second moments.
- AWQ
- Activation-aware scaling of salient input channels before rounding.
- Quantisation-aware training
- Training with rounding in the forward pass so the model adapts to it.
Sources
- Frantar, Ashkboos, Hoefler and Alistarh, “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers”, ICLR 2023.
- Lin et al., “AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration”, MLSys 2024.
- Hassibi and Stork, “Second Order Derivatives for Network Pruning: Optimal Brain Surgeon”, NeurIPS 1992.