Drills: Decoding.
Five questions at a time from the whole volume, until every sampler setting has a reason.
This plate draws rounds of five from the questions and predictions of the seven plates before it, with the reasoning and source plate after each answer.
The volume’s thread is that a model gives a distribution and decoding decides the text: maximising probability loops, sampling needs temperature and truncation, penalties and constraints act on tokens rather than meaning, and the probabilities are worth using because pretraining calibrates them.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Choose between search and sampling.
- Set temperature and truncation.
- Use penalties and constraints with care.
- Read calibration and stopping.
What to carry forward
Greedy and beam search find probable, repetitive text (plate XXII.1); temperature trades coherence against variety (plate XXII.2); truncation removes the tail, adaptively for top-p and min-p (plate XXII.3); repetition penalties break loops at a cost (plate XXII.4).
Constrained decoding makes output valid, not correct (plate XXII.5); pretrained probabilities are calibrated and post-training can spoil that (plate XXII.6); and length is set by an end token, a stop string or a cap (plate XXII.7). Vol. XXIII turns to the machines that run decoding for many users at once: serving.
Terms
- Beam search
- Keeping the w most probable partial outputs.
- Temperature
- Dividing logits by T.
- Top-p
- Sampling from the smallest set reaching probability p.
- ECE
- Expected calibration error.