Vol. XXII · Decoding Plate 7 of 8stopping, end of sequence, EOS, stop strings, max tokens, length, chat templatesAE

Stopping.

A model stops because it learned that text ends — or because something outside it says so.

Decoding needs a rule for when to stop, and there are three. The model can sample a special end-of-sequence token, which post-training teaches it to emit at the end of a turn (the chat templates of plate XVII.2 end every assistant message with one). The caller can pass stop strings — a newline, a closing tag, the label of the next speaker — and cut the output when one appears. And there is always a length cap, the max-tokens parameter of every API, which ends generation whatever the model was doing.

The course model micro (1.8 million parameters, four layers, trained on this course’s own plates in English and Chinese; plate I.9) runs in the page. micro was trained on continuous course text cut into windows and never saw its end-of-text token at the end of an answer, so it never emits it. The workshop stops it at the first full stop instead, and shows how temperature changes how long it takes to get there and how often it runs into the cap.

Below: the lengths, in tokens, of 24 samples ended at the first full stop or at the cap; readouts give the mean and the share that hit the cap.

What you will learn

Name the three ways generation stops.
Explain why stopping is learned.
See temperature change length.
Read a capped output as truncated.

Workshop

micro, prompt “The attention”, 24 samples with seeds 1–24, stopped at the first “.” token or at a cap of 40 tokens. Length: tokens before the stop. Capped: samples that reached 40 without a full stop.

Length is a decoding result

At temperature 1, micro’s samples run 19.7 tokens on average before the first full stop and 21% run out the cap of 40. Colder, at 0.4, the mean is 15.3 with 8% capped: the model’s most probable paths reach a full stop sooner. Hotter, at 1.3, it is 28.2 with 46% capped: each step has more chance of wandering off any sentence-ending path. The length of an answer is not a property of the model alone; it is set by the decoding.

A capped output is truncated, not finished, and APIs report it (a finish reason of “length” rather than “stop”); a program that treats it as complete will parse half a JSON object or half an answer. Models learn to stop from training data that ends, which is why base models ramble to the cap and chat models stop at the end of a turn. Reasoning models add a further stop inside the turn — the end of the thinking section — and a cap on it, the thinking budget, which is one of the test-time compute controls of plate XIX.10.

Stop strings are matched on text, not tokens, because a string such as “\n\nUser:” can be split across tokens in several ways; servers hold back the last few characters until they are sure no stop string is forming.

Terms

End-of-sequence token: A special token the model learns to emit at the end of its output.
Stop string: Text that ends generation when it appears.
Length cap: The maximum number of tokens to generate.
Finish reason: What ended generation: a stop, or the cap.

Generation stops at a learned end token, a stop string or a cap; a capped output is truncated, and decoding settings change how long text runs.

Sources

OpenAI API reference, “max_tokens” and “stop” parameters (accessed October 2026).
Hugging Face Transformers documentation, “Generation strategies” (eos_token_id, stopping criteria).