Bytes and Unicode.
Underneath every tokeniser is the same thing: text stored as numbers, one to four bytes per character.
A computer stores text as numbers. Unicode gives every character in every writing system a number, its code point: A is 65, é is 233, 光 is 20 809. UTF-8, the encoding almost all text on the web uses, writes each code point as a sequence of one to four bytes: the ASCII letters and digits take one byte, most European accented letters two, Chinese, Japanese and Korean characters three, and some rarer symbols four.
A tokeniser that starts from bytes has exactly 256 base symbols and can encode any text in any language — there is no character it has not seen, only byte sequences it has not merged. The cost is that languages whose characters take more bytes start from longer sequences, which is one reason the same content can cost different numbers of tokens in different languages (plate II.7).
Below: each character of the text with its code point and UTF-8 bytes. Type in any script; the bars show how many bytes each character takes.
- Look up the Unicode code point of a character.
- Encode code points as UTF-8 bytes.
- Count bytes per character in different scripts.
- Explain why byte-level vocabularies never meet an unknown symbol.
Variable width, universal coverage
The sample has 9 characters (counting the spaces) in 16 bytes, 1.78 bytes per character: the A, the x and the spaces take one byte each, é, π and the superscript two each, and 光 and the arrow three. UTF-8 was designed so that ASCII text is unchanged — an English file is the same bytes in ASCII and UTF-8 — and so that a byte tells you whether it starts a character or continues one, which lets a reader find character boundaries from any point. Those properties are why it won: by the 2020s nearly every web page used it.
For a language model the important consequences are two. First, 256 byte values cover everything, so a byte-level vocabulary has no unknown symbols: a tokeniser built on bytes, as GPT-2’s was in 2019 and the course tokeniser is, can always fall back to them. Second, a single token can contain part of a character. When the course tokeniser splits a rare Chinese character, a token may hold two of its three bytes; such a token is not a character at all and cannot be printed on its own, which is why the course writes it as <E5><8D>.
Terms
- Unicode
- The standard that assigns a number, a code point, to every character of every writing system.
- Code point
- The number Unicode gives a character, written U+ followed by hexadecimal digits.
- UTF-8
- The encoding that writes each code point as one to four bytes, ASCII unchanged.
- Byte-level vocabulary
- A vocabulary built up from the 256 byte values, so it can encode any text.