Vol. II · Tokens Plate 6 of 10vocabulary size, embedding parameters, softmax cost, compression, tied embeddingsAC

Choosing a Vocabulary Size.

Every token added shortens sequences a little and costs a row of parameters at both ends of the model.

A vocabulary of V tokens costs the model an embedding table of V × d numbers at its input and, unless the two are tied, an output matrix of the same size that produces one logit per token at every position. For a small model these tables can be most of its parameters; for a large one they are a few per cent. On the other side of the scale, every extra token shortens sequences a little — but less and less, because the merges that save most come first (plate II.3).

So the right vocabulary size depends on the model. A small model wants a small vocabulary, so as not to spend its parameters on embeddings of tokens it will rarely see; a large model can afford a large one, and multilingual models need one, because every language’s common words and characters must fit. The course models use 8 192 tokens; GPT-2 used 50 257; Llama 3 uses 128 256.

Below: left, how well the course tokeniser compresses English and Chinese when cut to its first 512, 1 024, … 8 192 tokens. Right, the embedding and output tables against the rest of the model for the configuration chosen — the course model micro, GPT-2 small, or the published Llama 3 8B configuration.

What you will learn
Workshop
Left: characters per token for the English and Chinese corpus slices with the course tokeniser cut to 512–8 192 tokens. Right: parameters in the vocabulary tables against the rest of the network for the chosen published configuration.

Rows at both ends

Cutting the course tokeniser back to its first 512 tokens shows the curve of diminishing returns: at 512 tokens English averages 1.78 characters per token and at 8 192 it averages 3.58, a gain of 2.02 times for sixteen times the vocabulary; Chinese goes from 0.63 to 1.47, starting below one character per token because a Chinese character is three bytes. On the other side, the course model micro, with tied embeddings and width 128, spends 1.05 million of its 1.85 million parameters — 57% — on the vocabulary table. GPT-2 small, tied, at width 768, spends 38.6 million, 31%. The published Llama 3 8B configuration, untied, with 128 256 tokens at width 4 096, spends 1.05 billion, 13%: a large model can afford a vocabulary that a small one cannot.

Tying the input embedding to the output matrix (plate V.7) halves the cost and was standard in GPT-2-era models; many large models now leave them untied, since the extra parameters are a small share for them. The output logits also cost compute — about 2 V d floating-point operations per token, 2.1 million for the chosen configuration — and memory during training, where a logit tensor of batch × length × V numbers can be one of the largest activations. And every token needs enough training data to learn a useful row: with too large a vocabulary for its data, a model ends up with rare, barely trained tokens (plate II.8).

The Llama 3 report gives the move from 32 000 to 128 000 tokens as the source of better compression and multilingual coverage; models in 2025 ranged from about 32 000 to over 250 000 tokens.

Terms

Embedding table
The V × d matrix whose rows are the input vectors of the tokens.
Output matrix
The d × V matrix that turns the final hidden state into a logit per token.
Tied embeddings
Using the same matrix for input embeddings and output logits.
Diminishing returns
Each added token shortens sequences less than the one before.
A vocabulary costs V × d parameters at each end and returns ever smaller gains in compression. Small models want small vocabularies; large and multilingual models can afford large ones.

Sources

  1. GPT-2 small: Radford et al., “Language Models are Unsupervised Multitask Learners”, OpenAI, 2019.
  2. Llama 3 8B: Llama Team, AI @ Meta, “The Llama 3 Herd of Models”, arXiv:2407.21783, 2024, Table 3.