Drills: Tokens.
Five questions at a time from the whole volume, until a tokeniser holds no surprises.
This plate draws rounds of five from the questions and predictions of the nine plates before it, with the reasoning shown after each answer and the plate it came from.
Tokenisation is the part of a language model most often ignored and most often responsible for strange behaviour. Work through rounds until the arithmetic of bytes, merges and vocabulary tables is automatic.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Recall how BPE trains and encodes.
- Compute bytes, tokens and embedding costs quickly.
- Predict tokenisation artefacts.
- Explain the role of special tokens.
What to carry forward
A model reads token ids from a fixed vocabulary, built here by byte-pair encoding: bytes first, then the most frequent pairs merged one by one, applied word by word to new text (plates II.2–II.4). Vocabulary size trades shorter sequences against embedding rows at both ends of the model (plate II.6), and the tokeniser’s training data decides what each language costs (plate II.7).
Tokenisation also explains a family of failures — irregular numbers, hidden letters, trailing spaces, untrained tokens (plate II.8) — and special tokens carry the structure that conversations need (plate II.9). The next volume opens up the network that reads these ids: embeddings, losses, backpropagation and optimisers.
Terms
- BPE
- Byte-pair encoding: merge the most frequent adjacent pair, repeatedly.
- Pre-tokenisation
- Splitting text into words before merging.
- Vocabulary table
- V × d parameters at the input and, untied, at the output.
- Special token
- A vocabulary entry for structure, never produced by text.