Vol. XXII · Decoding Plate 2 of 8temperature, sampling, softmax, entropy, diversity, logitsAE

Temperature.

Divide the logits by T before the softmax: sharper below one, flatter above.

Sampling picks the next token at random in proportion to its probability, so it writes text with the variety of the training data rather than the monotony of the mode. Temperature adjusts the trade-off: the logits are divided by T before the softmax, p_i ∝ exp(z_i / T). At T → 0 sampling becomes greedy; at T = 1 it samples the model’s own distribution; as T grows it approaches choosing uniformly among all tokens. The name comes from statistical physics, where the same formula gives the Boltzmann distribution.

The course model micro (1.8 million parameters, four layers, trained on this course’s own plates in English and Chinese; plate I.9) runs in the page. The workshop samples six 30-token continuations at the chosen temperature and measures how probable the model itself finds them and how repetitive they are.

Below: one sampled continuation at the chosen temperature; readouts give the mean log-probability per token under micro at temperature 1 and the share of distinct two-token pairs, averaged over six samples.

What you will learn
Workshop
micro, prompt “The attention”, six samples of 30 tokens with seeds 1–6. Log-probability per token: under micro’s own distribution at temperature 1, so it measures how typical the text is to the model. Distinct 2-grams averaged over the six.

Between loops and noise

At temperature 1 the samples average −4.35 nats per token under micro — as uncertain as the model’s own predictions, by construction — and 99% of their two-token pairs are distinct. At 0.3 they are far more probable (−1.75) and more repetitive (72% distinct): sampling is close to greedy. At 1.8 they average −8.68: each token is chosen from a flattened distribution in which thousands of unlikely tokens together win often, and the text stops making sense even locally.

The useful range is narrow and depends on the task. Factual answers and code are usually sampled cold (0 to 0.7) because one good answer is wanted; creative writing and brainstorming warmer (0.7 to 1.0); sampling many candidates for a verifier or vote (plate XIX.10) warmer still, because variety is the point. Temperature alone cannot fix the tail problem — at any T above zero, every token in the vocabulary keeps some probability — which is what truncation (plate XXII.3) addresses.

Temperature changes the ranking of nothing: the most probable token stays most probable at every T. It changes only how decisively the ranking is followed.

Terms

Sampling
Choosing the next token at random according to its probability.
Temperature
Dividing logits by T to sharpen (T < 1) or flatten (T > 1) the distribution.
Entropy
The spread of the distribution, which rises with temperature.
Tail
The many low-probability tokens of the vocabulary.
Temperature divides the logits by T: low T approaches greedy loops, high T dissolves into the tail.

Sources

  1. Ackley, Hinton and Sejnowski, “A Learning Algorithm for Boltzmann Machines”, Cognitive Science 9, 1985 (temperature in sampling).
  2. Holtzman et al., ICLR 2020.