Generating Text.
A model that predicts the next token can write: pick one, append it, and predict again. How it picks changes everything.
A language model writes one token at a time. Given a prompt, it returns the next-token distribution; a decoding rule chooses one token from it; that token is appended to the text and the model is asked again. Because each output becomes part of the next input, generation is called autoregressive. The model itself never changes during this loop — only the text grows.
The obvious rule is to take the most probable token every time: greedy decoding. It produces the single most likely continuation step by step, and it tends to fall into loops: once a phrase has been produced, repeating it becomes the most likely thing to do. The other basic rule is to sample: pick each token at random in proportion to its probability. A temperature divides the logits before the softmax; below one it sharpens the distribution towards greedy, above one it flattens it towards noise.
Below: the course model micro continues a prompt for 48 tokens. Temperature 0 is greedy. Press “Sample again” to draw another continuation at the same temperature; the lower panel shows the probability of each chosen token.
- Describe autoregressive generation as a loop over the next-token distribution.
- Compare greedy decoding with sampling.
- Explain why greedy text repeats itself.
- Use temperature to trade variety against probability.
Most likely is not best
Greedy decoding, temperature 0, gives the continuation whose tokens the model rated most probable one at a time — mean probability 0.347 per chosen token — and 54% of its three-token sequences are repeats of earlier ones: the text loops. Raise the temperature to 0.8 and sample: the mean probability of the chosen tokens falls to 0.229, the repetition falls to 0%, and the text wanders like the corpus it was trained on. At 1.4 almost nothing repeats (0%), but the model is now often choosing tokens it rated unlikely and the text dissolves.
That is the basic trade-off of decoding, and Vol. XXII returns to it with better rules (top-k, top-p, min-p, beam search, penalties). The lesson for now is that “what the model thinks” and “what the model writes” are different things: one model, one distribution, and very different text depending on the rule that turns the distribution into tokens. Large models loop far less than this small one, but greedy decoding still produces their blandest and most repetitive text.
Terms
- Autoregressive generation
- Producing text one token at a time, each chosen token fed back as input for the next prediction.
- Greedy decoding
- Always choosing the most probable next token.
- Sampling
- Choosing the next token at random, with probability proportional to the model’s distribution.
- Temperature
- A number dividing the logits before the softmax: below 1 sharpens the distribution, above 1 flattens it.