Truncation.
Throw away the tail, then sample. How much to throw away is the difference between the methods.
At temperature 1 the tail of an 8 192-token vocabulary holds a few per cent of probability spread across thousands of tokens, almost all of them wrong in context; over a long text, sampling from it is near certain to happen many times. Top-k sampling (Fan et al., 2018) keeps only the k most probable tokens. Nucleus or top-p sampling (Holtzman et al., 2020) keeps the smallest set whose total probability reaches p, so it keeps few tokens when the model is confident and many when it is not. Min-p (Nguyen et al., 2024) keeps every token whose probability is at least a fraction of the most probable one’s, which scales the cut with confidence the same way.
The course model micro (1.8 million parameters, four layers, trained on this course’s own plates in English and Chinese; plate I.9) runs in the page. The workshop samples with the chosen rule and shows how many tokens it keeps at the first step, how probable the samples are under the model, and how repetitive.
Below: a sample with the chosen truncation at temperature 1, and the number of tokens kept at the first step; readouts compare the samples’ log-probability and variety across rules.
- Define top-k, top-p and min-p.
- Explain why the tail derails sampling.
- Compare how each rule adapts to confidence.
- Choose a rule and a setting.
Cut the tail, keep the head
Untruncated, samples at temperature 1 average −4.35 nats per token. Keeping only the top 10 tokens raises that to −2.61; nucleus sampling at p = 0.9 to −3.38, keeping 668 tokens at the first step after this prompt, where micro is unsure; min-p at 0.1 to −2.68, keeping 17. Every rule makes the samples more probable under the model by removing tokens that are each unlikely and collectively frequent.
They differ in how they adapt. Top-k keeps the same number whether the model is sure or unsure: too many when one token is obvious, too few when many are reasonable. Top-p and min-p scale with the shape of the distribution, which is why top-p is the usual default that serving APIs expose; min-p was proposed for higher temperatures, where top-p’s cumulative cut lets in more of a flattened tail. All of them can be combined with temperature, and in practice are: a typical chat setting is temperature 0.7 with top-p 0.9.
Terms
- Top-k
- Sampling only among the k most probable tokens.
- Top-p (nucleus)
- Sampling from the smallest set whose probability reaches p.
- Min-p
- Keeping tokens with probability at least m times the top token’s.
- Truncation
- Removing the low-probability tail before sampling.
Sources
- Fan, Lewis and Dauphin, “Hierarchical Neural Story Generation”, ACL 2018 (top-k).
- Holtzman et al., ICLR 2020 (nucleus sampling).
- Nguyen et al., “Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs”, ICLR 2025.