Vol. XXXI · History and Projects Plate 6 of 14project, tokeniser, BPE, compression, vocabulary size, training textD

Project: A Tokeniser.

Choose how many merges to learn and how much text to learn them from, then measure on text the tokeniser never saw.

The brief: a byte-level BPE tokeniser (Vol. II) for this course’s English text that encodes the last 12 000 characters of the corpus — never used in training — at no fewer than 2.58 bytes per token, with at most 1 300 vocabulary entries (256 bytes plus the merges), because every entry costs an embedding row and a column of the output layer.

Two settings: the number of merges and the amount of training text, taken from the start of the corpus. The starting design learns 500 merges from 20 thousand characters and compresses the held-out text to 2.18 bytes per token. A project plate grades the design against a brief: every check below the workshop must pass, and a passing design is saved in this browser.

Left: bytes per token on the held-out text for the current design, against the brief’s line. Right: the first tokens of a held-out sentence as the tokeniser splits it.

What you will learn

Train a BPE tokeniser to a target.
Trade vocabulary size against compression.
See the effect of training-text size.
Measure on held-out text.

Workshop

Byte-level BPE as in plate II.3, trained on the first characters of the course’s English corpus (up to 140 000), evaluated on characters 150 000–162 000. Merges are learned greedily by pair frequency.

Two budgets that pull apart

More merges compress better but enlarge the vocabulary; more training text makes the merges that are learned more representative of text in general. The starting design fails on compression, at 2.18 bytes per token: 500 merges cannot capture enough of English’s frequent words. Raising the merges alone runs into the vocabulary cap before it reaches the target on little text, because merges learned from 20 thousand characters fit that text’s accidents — names and phrases that recur there and nowhere else.

The general lesson is the one of plates II.6–7: a tokeniser is a model of its training text, and judged, like any model, on text it did not see. Production tokenisers make the same trade at a larger scale — vocabularies of 100 000 to 260 000 entries trained on samples of the pre-training mix, chosen against the cost of the embedding and output layers and the compression each language gets.

The held-out text here comes from the same course, so it is easier than truly new text would be; a tokeniser trained on English course text compresses Chinese, or code, far worse (plate II.7).

Terms

Merge: A learned rule joining two adjacent tokens into one.
Bytes per token: How much text each token carries on average.
Held-out text: Text not used for training, used to measure.
Vocabulary: All tokens: the 256 bytes plus every merge.

A tokeniser meets a compression target by enough merges learned from enough text, within a vocabulary budget, measured on text it never saw.

Sources

Sennrich, Haddow and Birch, “Neural Machine Translation of Rare Words with Subword Units”, ACL 2016.
Radford et al., “Language Models are Unsupervised Multitask Learners”, OpenAI, 2019 (byte-level BPE).