Vol. XVII · Supervised Fine-Tuning Plate 2 of 10chat template, special tokens, roles, system prompt, end of turn, ChatML, Llama 3 template, prompt formatAE

Chat Templates.

System, user, assistant: three roles flattened into one stream of tokens, with markers no user can type.

A Transformer reads one sequence of tokens, so a conversation must be flattened. A chat template does it: each turn is wrapped in a header naming its role — system, user or assistant — and closed with an end-of-turn token, and the prompt ends with an open assistant header so the model’s next tokens are the reply. Generation stops when the model emits the end-of-turn token. OpenAI’s ChatML uses <|im_start|> and <|im_end|>; Llama 3 uses <|start_header_id|>, <|end_header_id|> and <|eot_id|>.

The markers are special tokens: single entries in the vocabulary that the tokeniser produces only from the template, never from user text, so a user cannot forge the start of an assistant or system turn by typing the characters (plate II.10). Fine-tuning binds the model’s behaviour to these exact markers. The workshop renders one conversation in three templates and counts its tokens with the markers as special tokens and as plain text.

Below: the conversation flattened by the chosen template, token by token, with the role markers highlighted. Readouts compare token counts with the markers as special tokens and as plain text.

What you will learn
Workshop
One system message, one user message and one assistant reply, with an open assistant header at the end, rendered in ChatML, the Llama 3 template and the course’s own “U:…|A:…|” format. Tokens from the course BPE tokeniser (plate II.6); markers either registered as special tokens (one token each) or left as plain text.

Markers no one can type

In the Llama 3 template this short conversation is 52 tokens, 12 of them role and turn markers; written out as plain text, the same markers would cost 107 extra tokens and — more importantly — would be ordinary text that any user could type. ChatML’s version is 46 tokens; the course’s compact format, which uses printable characters as markers, is 29, and it has exactly that weakness: a user who types “|A:” in a message forges an assistant turn. Production templates reserve markers that the tokeniser will only produce from the template itself.

The template is part of the model. A model fine-tuned on Llama 3’s headers and prompted in ChatML sees an unfamiliar sequence and falls back towards base-model behaviour: continuing text, ignoring the system message, failing to stop. That is why libraries ship each model’s template alongside its weights, and why “the model is broken” is often “the prompt is in the wrong format”. The system turn is where deployments put their standing instructions; whether the model follows it is learned in fine-tuning, not guaranteed by the format.

Some templates differ only in whitespace — a newline after a header or not — and even that can change a fine-tuned model’s output, because the newline is a token the model always saw in training.

Terms

Chat template
The rule that flattens a conversation into one token sequence.
Role header
The marker naming a turn as system, user or assistant.
End-of-turn token
The special token that closes a turn and stops generation.
System prompt
Standing instructions placed in the first turn.
A chat template flattens roles and turns into one token stream with special-token markers; the model only behaves as fine-tuned when prompted in the same template.

Sources

  1. Meta, Llama 3 model card and prompt format documentation, 2024.
  2. OpenAI, ChatML documentation, 2023.
  3. Hugging Face Transformers documentation, “Chat templates”.