Vol. II · Tokens Plate 7 of 10multilingual tokenisation, token parity, fairness, Chinese, code, compressionDC

The Cost of a Language.

A tokeniser trained on English charges other languages extra: more tokens for the same meaning, which means more compute, less context and higher prices.

A model processes, prices and limits text in tokens. If the same paragraph takes 1 000 tokens in English and 2 500 in another language, a reader of the second language pays two and a half times as much per question, fits two and a half times less into the context window, and waits longer for each answer. Whether that happens depends almost entirely on the tokeniser’s training data: frequent languages get long tokens, rare ones are spelled in bytes.

The course corpus is parallel: every plate of every Curionode course exists in English and in Chinese with the same content. That makes it a fair test. The workshop takes the same twenty plates in both languages and tokenises them with the course tokeniser, trained on both, and with a BPE tokeniser trained in your browser on English alone.

Below: twenty plates of the course corpus, the same content in English and in Chinese, counted in bytes, characters and tokens under two tokenisers. Pairs of bars compare the languages; the ratio is the extra cost of Chinese.

What you will learn

Measure the token cost of the same content in two languages.
Explain how a tokeniser’s training data sets each language’s cost.
Relate token counts to price, speed and context length.
Recognise why multilingual models need multilingual tokenisers.

Workshop

Twenty parallel plates of the course corpus: English (blue) and Chinese (red) measured in UTF-8 bytes, characters, course tokens and tokens of an English-only BPE trained in the page.

Who pays for the vocabulary

The Chinese text needs only 0.39 times as many characters as the English but, because each character is three bytes, 0.95 times as many bytes. The course tokeniser, trained on both languages, needs 15,891 tokens for the English and 14,759 for the Chinese: a ratio of 0.93, near parity. The BPE trained in the page on English alone needs 19,467 tokens for the English but 51,570 for the Chinese, 2.65 times as many — it has hardly any merges for Chinese bytes, so most of the Chinese text is encoded byte by byte. Same content, same meaning, 2.6 times the cost.

Published studies of production tokenisers found the same pattern at scale: in 2023, Petrov and colleagues measured the same sentences in many languages and found some needing several times more tokens than English, and up to about fifteen times for the worst-served. The effects compound. Prices are per token; the context window is a fixed number of tokens; generation time is per token; and a model trained on fewer tokens of a language has also seen less of it. Multilingual tokenisers with larger vocabularies narrowed the gap considerably in later models. The same arithmetic applies to code, whose indentation and symbols a prose-trained tokeniser handles badly: the course tokeniser spends one token on each space of an indent.

Because the course corpus is translated plate by plate, it is unusually fair as a test: each Chinese plate says what its English twin says, so a difference in tokens is a difference in tokenisation, not in content.

Terms

Token parity: The same content costing the same number of tokens in every language.
Byte fallback: Encoding text the tokeniser has no merges for one byte at a time.
Context window: The maximum number of tokens a model reads at once; fewer tokens per sentence fit more meaning.
Parallel corpus: The same content in two or more languages, aligned piece by piece.

A tokeniser charges each language according to its training data. Trained on English alone, it made the same Chinese content cost 2.6 times as many tokens; trained on both, about the same.

Sources

Petrov, La Malfa, Torr and Bibi, “Language Model Tokenizers Introduce Unfairness Between Languages”, NeurIPS 2023.