Special Tokens and Templates.
A handful of tokens never appear in ordinary text. They tell the model where a document ends and, in an assistant, who is speaking.
Some things a model must know are not in the text: where one training document ends and the next begins, which positions are padding, and — once a model becomes an assistant — which words were written by the user and which by the model. These are marked with special tokens: vocabulary entries that the tokeniser never produces from ordinary text and that are inserted by the software around the model. The course tokeniser has one, token 0, <|endoftext|>, placed between documents during pretraining.
A chat model is trained on conversations laid out with a chat template: each turn is wrapped in special tokens that name its role — system, user, assistant — and the model learns to produce the assistant’s turn and then the token that ends it. The template is part of the model’s training data and must be reproduced exactly when the model is used; a template off by one token degrades the answers. Vol. XVII returns to templates when the course model is fine-tuned.
Below: a short conversation laid out with a chat template, tokenised with the markers registered as special tokens, and without. Underneath, the probability micro gives the end-of-text token after a complete sentence and in the middle of one.
- Explain what special tokens are for and how they differ from text.
- Lay out a conversation with a chat template.
- See why role markers must be single tokens that users cannot type.
- Measure a base model’s learned sense of where documents end.
Structure the text cannot carry
With its markers registered as special tokens, the conversation is 29 tokens, of which 6 are structure; spelled out as ordinary text, the same markers cost 34 tokens and become strings like <|, user and |> that any user could type into a message. That is the real reason markers must be special: a role boundary that can be typed can be forged, and text pretending to be a system or assistant turn is a classic way to confuse an assistant about who said what. Serving software therefore inserts special tokens itself and refuses, or escapes, their text forms in user input; Vol. XXV returns to this as a defence.
The one special token the course model has already learned is <|endoftext|>. After a finished sentence micro gives it a probability of 3.6 × 10⁻²; in the middle of a phrase, 2.0 × 10⁻⁵, about 1.7 × 10³ times less. Nobody told it where documents end: it learned it from where the token was inserted in its training data. A chat model learns the end of its turn the same way, which is how it knows when to stop talking (plate XXII.7). Other special tokens in common use mark the beginning of a sequence, padding in batches of unequal length, masked positions in encoder models, images and audio in multimodal models (Vol. XXIX), and tool calls in agents (Vol. XXV).
Terms
- Special token
- A vocabulary entry that ordinary text never produces, inserted by software to mark structure.
- End-of-text (EOS)
- A token marking the end of a document or of a model’s turn.
- Chat template
- The exact layout of roles, turns and special tokens a chat model was trained on.
- Padding
- Filler positions that make sequences in a batch the same length, masked out of attention and loss.