The Unigram Tokeniser.
Give every piece a probability, choose the segmentation of each word with the highest total, and prune the pieces that matter least.
BPE builds a vocabulary bottom-up, one merge at a time, and encodes greedily. The unigram tokeniser, introduced by Taku Kudo in 2018 and used in SentencePiece, works the other way. It starts from a large set of candidate pieces — every frequent substring — and gives each a probability. A word is segmented into the sequence of pieces whose probabilities multiply to the most, found with the Viterbi algorithm. Training alternates two steps: segment the corpus with the current probabilities and re-estimate each piece’s probability from how often it was used; then drop the pieces whose removal would cost the least, until the vocabulary has the size wanted.
Because it keeps probabilities, a unigram tokeniser can also sample segmentations rather than always take the best one — a regulariser that shows a model the same word cut in different ways. Several widely used models, including T5 and Gemma, have used SentencePiece unigram vocabularies; most recent large models use byte-level BPE.
Below: a unigram tokeniser and a BPE tokeniser trained in your browser on the same 60 000 characters with the same vocabulary size, compared on held-out text and on a sample sentence. The curve shows the unigram vocabulary being pruned round by round.
- Segment a word by maximising the product of piece probabilities.
- Describe unigram training as estimation followed by pruning.
- Compare unigram and BPE segmentations at the same vocabulary size.
- Explain subword sampling as a regulariser.
Top-down instead of bottom-up
Both tokenisers here work on characters rather than bytes and were trained on the same 60 000 characters. At 800 pieces the unigram tokeniser encodes the next 20 000 characters in 8,794 tokens, 2.27 characters each, and BPE in 8,684, 2.30 each; training took 4 rounds of estimating and pruning. At 400 pieces the counts are 10,512 and 10,090; at 2 000, 7,372 and 7,207. The two methods compress about equally well. Where they differ is in the shape of the cuts: unigram prefers segmentations made of a few high-probability pieces and tends to find morpheme-like boundaries, while greedy BPE can leave odd fragments where its earliest merges happened to fire.
In practice the choice matters less than the data and the vocabulary size. What matters more for a byte-level model is that unigram, as originally designed, works on characters, so it needs a fallback for characters never seen in training — SentencePiece adds byte fallback for this — while byte-level BPE covers everything by construction. This simplified unigram trainer seeds its candidates with frequent substrings, estimates probabilities from Viterbi counts instead of the full expectation, and prunes a fifth of the pieces per round; Kudo’s original does the expectation exactly and prunes by the loss each removal would cause.
Terms
- Unigram tokeniser
- A tokeniser that gives every piece a probability and chooses the most probable segmentation.
- Viterbi algorithm
- Dynamic programming that finds the best segmentation of a word in time proportional to its length.
- Pruning
- Removing the pieces whose loss would cost least, until the vocabulary has the wanted size.
- Subword regularisation
- Sampling different segmentations of the same text during training.