Why Not Characters or Words.
Every way of cutting text into symbols trades the length of the sequence against the size of the vocabulary.
Before a model can read a text it has to be turned into a sequence of symbols from a fixed list, the vocabulary. The two obvious choices both fail. Characters (or bytes) give a tiny vocabulary that covers every text, but sequences become long: the model has to spend a step on every letter, and the cost of attention grows with the square of the length (Vol. IV). Whole words give short sequences but an enormous vocabulary — English alone has hundreds of thousands of word forms, and every name, typo and new coinage is a word the model has never seen.
Modern models use subword tokens: common words are single tokens, rare words are spelled out of a few common pieces, and anything at all can still be written as bytes. The course tokeniser has 8 192 tokens; production models use 32 000 to about 260 000. This volume shows how such a vocabulary is built and what it does to the model that reads it.
Below: one sentence cut four ways — into characters, UTF-8 bytes, words and the course tokeniser’s subwords. Type your own text; the bars compare how many symbols each needs and how much text each covers on average.
- Explain the trade-off between sequence length and vocabulary size.
- Count tokens under character, byte, word and subword schemes.
- See why word vocabularies cannot cover new text.
- Relate sequence length to the cost of a forward pass.
Length against vocabulary
The sample sentence has 89 characters and 97 bytes — more bytes than characters because the Chinese characters take three bytes each in UTF-8. Split into words and punctuation it is 15 symbols, and in the course tokeniser’s subwords 31, about 2.87 characters per token. The subword split keeps common words whole, breaks unbelievably into pieces it has seen, cuts the long number into chunks of digits, and spells the Chinese term out of characters and pairs. Nothing is ever out of vocabulary, because a byte-level tokeniser can fall back to single bytes.
Why not just use bytes and avoid the question? Because sequence length is expensive. A forward pass costs work in proportion to the number of tokens, and attention in proportion to its square; a byte-level model of the same quality would have to process three or four times as many positions, and its context window would hold three or four times less text. Why not whole words? Because the vocabulary would have to be huge — every word form needs its own embedding row and its own output logit (plate II.6) — and it would still fail on the first name it had never seen. Subwords are the compromise that every large model in use today makes; how they are chosen is the subject of plates II.3 to II.5.
Terms
- Vocabulary
- The fixed list of symbols a model reads and predicts.
- Subword token
- A piece of a word: common words are whole tokens, rare words are spelled from several.
- Out of vocabulary
- A word or symbol a tokeniser cannot represent; byte-level tokenisers never have one.
- Sequence length
- The number of tokens a text becomes; a model’s cost grows with it.