Drills: Cheap Inference.
Five questions at a time from the whole volume, until every saving has a price you can name.
This plate draws rounds of five from the questions and predictions of the nine plates before it, with the reasoning and source plate after each answer.
The volume’s thread is that decode cost is bytes read per token: fewer bits per weight, fewer weights, fewer cached bits, or fewer passes per token — each bought with some accuracy, some memory or some engineering.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Choose a quantisation scheme.
- Reason about speculative decoding.
- Weigh pruning, distillation and MoE.
- Size a model for a device.
What to carry forward
Round-to-nearest is free at 8 bits and cheap at 4 with groups (plate XXIV.1); GPTQ and AWQ minimise output error on calibration data, and GPTQ can overfit it (plate XXIV.2); quantised caches multiply concurrency (plate XXIV.3); speculative decoding skips passes with exact outputs (plate XXIV.4), and its drafts can come from the model or the prompt (plate XXIV.5).
Distilled small models keep much of the quality at a fraction of the bytes (plate XXIV.6); pruning by weight times input beats magnitude, and only hardware-friendly patterns save time (plate XXIV.7); MoE models need very large batches (plate XXIV.8); and on devices bandwidth is the budget (plate XXIV.9). Part Six turns from the model to what it does in the world: agents.
Terms
- Quantisation
- Fewer bits per number.
- Speculative decoding
- Draft, then verify in one pass.
- Wanda
- Pruning by |w| times input norm.
- Active parameters
- Those used per token.