When Tokens Get in the Way.
A model never sees letters or digits unless a token happens to be one. Many of its odd failures start here.
A model reads token ids. When a word is a single token, the model has no direct view of its letters — asked how many r’s are in strawberry, it must have learned the spelling of that token from somewhere, because it cannot look. Numbers are cut into chunks that depend on the tokeniser’s merges, so 1234 and 12345 may be split in unrelated ways, and the digit in the hundreds place is in a different token position from one number to the next. Arithmetic is harder than it needs to be.
Two subtler effects follow from how the vocabulary was built. A prompt that ends in a space puts the model in a state it rarely saw in training, because in training text a space almost always belongs to the next token: it then expects the second half of a word. And tokens that were frequent in the tokeniser’s training data but rare in the model’s are barely trained — the “glitch tokens” of 2023 were of this kind. The course tokeniser and micro show all four.
Below: numbers and a word in the course tokeniser; the next-token predictions of micro for a prompt with and without a trailing space; and the embeddings of the tokens micro saw fewer than ten times in training, compared with frequent ones.
- See how numbers and words are split into tokens.
- Explain why trailing spaces change predictions.
- Identify under-trained tokens from training counts and embeddings.
- Connect tokenisation to failures in spelling and arithmetic.
Four ways tokens leak through
Numbers first: in the course tokeniser ·1234 is 2 tokens and ·12345 is 3, cut at different places, so the model never sees digits aligned by place value; tokenisers that split digits one by one or into fixed groups of three make arithmetic noticeably easier to learn. Spelling: strawberry and Strawberry are different token sequences, and a model learns a token’s letters only indirectly, from text that happens to spell it out. A trailing space: after the prompt without one, micro’s distribution has an entropy of 7.48 bits and its favourite tokens are whole words; add a space and the entropy is 4.69 bits with its favourites being word endings — in training, a space was almost always the start of the next token, so a prompt ending in one looks like a word cut in half.
Under-trained tokens: 216 of the course tokeniser’s 8 192 tokens occurred fewer than ten times in micro’s training ids (151 never), mostly bytes that never appear in valid UTF-8 and pieces that are always swallowed by longer merges. Their embedding rows are almost interchangeable — a mean cosine similarity of 0.995 to their group’s mean, against 0.411 for frequent tokens — because the only training signal they ever received was the push, at every position, to make them less likely. Feed one to the model and it cannot tell it from any other. In 2023 users found tokens such as SolidGoldMagikarp that made production models behave strangely for exactly this reason: they were in the tokeniser, built from different data, but almost absent from the model’s training text. Methods that scan a model’s weights for such tokens now exist and found them in most open models examined.
Terms
- Glitch token
- A token that makes a model behave strangely because it was almost never seen in training.
- Under-trained token
- A vocabulary entry whose embedding received too little training signal to mean anything.
- Digit splitting
- Tokenising numbers digit by digit, or in fixed-size groups, so place value is visible.
- Token boundary
- Where one token ends and the next begins; prompts that end mid-token confuse models.
Sources
- Land and Bartolo, “Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models”, EMNLP 2024.