Vol. I · What a Language Model Is Plate 4 of 10cross-entropy, perplexity, bits per byte, held-out evaluation, compressionE

Perplexity and Bits.

How good is a language model? Average its surprise over text it has never seen — and divide by bytes, not tokens, if the tokenisers differ.

Plate I.3 scored one text. To grade a model, score a lot of text that was held out of training and average the surprisal per token: that average is the cross-entropy of the model on the data, in nats or bits per token. Its exponential is the perplexity: a cross-entropy of 3 bits per token is a perplexity of 8, as if at each step the model were choosing among eight equally likely tokens.

Per-token numbers have a trap: they depend on what a token is. A character-level model predicts many easy tokens; a model with a large vocabulary predicts fewer, harder ones. Dividing the total surprisal by the number of bytes of text gives bits per byte, which is fair across tokenisers — and is also, exactly, the size the text would take if the model were used as a compressor.

Below: five kinds of text scored by the course model micro. The left panel shows the surprisal of each token of the chosen text; the right compares bits per byte across all five.

What you will learn

Compute cross-entropy and perplexity from per-token surprisal.
Convert between nats, bits, perplexity and bits per byte.
Explain why bits per byte is fair across tokenisers.
Relate a model’s loss to how well it would compress text.

Workshop

Left: surprisal in bits of each token of the chosen text under micro. Right: bits per byte for all five texts — what each would cost to store if micro were the compressor.

One number, three units

On the English sentence, micro averages 4.97 bits per token — a perplexity of 31.2 — and 1.185 bits per byte. The Chinese sentence costs 5.45 bits per token and 1.258 bits per byte: a Chinese character takes three bytes of UTF-8, so its tokens cover more bytes than English ones, and the per-byte figures show the two languages closer together than the per-token figures do. The repeated pattern starts expensive and gets cheaper each round — its first token costs 11.7 bits, its last 2.2 — but a model this small copies from its context poorly, so on average it pays 1.106 bits per byte, hardly less than prose; a large model finds the later rounds almost free (plate XXVIII.4 shows the circuit that does it). The random letters cost 6.851, more than the log₂ 26 ≈ 4.7 bits a random letter really carries, because the model keeps expecting English. The digits of π cost 4.181; each digit carries log₂ 10 ≈ 3.32 bits, and no model can pay less than that on π without having memorised the digits.

Bits per byte is also a compression ratio. Arithmetic coding turns any model into a lossless compressor that stores a text in almost exactly its total surprisal; at 1.185 bits per byte, micro would store the English sentence in about 1.185 / 8 of its original size. This is why “language modelling is compression” is more than a slogan, and why Vol. VIII reports the course models’ losses in bits per byte when tokenisers differ. Remember that these are single sentences: a real evaluation averages over millions of held-out tokens, and Vol. XXI adds the error bars.

Perplexity entered speech recognition in the 1970s, when Frederick Jelinek’s group at IBM needed a single number for how hard a recognition task’s language was; the term itself comes from information theory’s description of a distribution as uncertain as a fair die with that many faces.

Terms

Cross-entropy: The average surprisal per token of a model on some data; the loss a language model is trained to minimise.
Perplexity: The exponential of the cross-entropy: the number of equally likely choices with the same uncertainty.
Bits per byte: Total surprisal in bits divided by the number of bytes of text; comparable across tokenisers.
Held-out data: Text kept out of training and used only to measure the model.

Cross-entropy, perplexity and bits per byte are one measurement in three units. Compare models in bits per byte when their tokenisers differ, and always on text they were not trained on.