Vol. VIII · The Pretraining Run Plate 4 of 10perplexity, bits per byte, bits per character, nats, tokeniser comparison, baselines, compressionLE

What a Loss Value Means.

Nats per token depend on the tokeniser; bits per byte do not — the fair unit for comparing language models.

Loss in nats per token has two common translations. Perplexity, e to the power of the loss, is the number of equally likely choices that would cause the same surprise: a loss of 3.9 nats means the model is, on average, as uncertain as if it were picking among about 50 tokens. And because a loss is a code length (plate I.4), it can be expressed as bits per byte: total bits for a text divided by its length in UTF-8 bytes.

Bits per byte matter because losses per token cannot be compared across tokenisers. A character-level model like nano predicts more, easier units; a BPE model like micro predicts fewer, harder ones; their per-unit losses differ by a factor of almost three for reasons that have nothing to do with quality. Divided by bytes, both become the cost of encoding the same text, and the comparison is fair. The workshop scores the same held-out passage with both models.

Below: the same held-out passage scored by nano and by micro: loss per unit, perplexity, and bits per byte; and the baselines of a uniform guess over bytes and over micro’s vocabulary.

What you will learn

Convert loss to perplexity.
Convert loss to bits per byte.
Compare models with different tokenisers fairly.
Place a loss between its baselines.

Workshop

One held-out English passage, scored by nano (characters) and micro (BPE tokens). Left: bits per byte for a uniform guess over 256 byte values, a uniform guess over micro’s 8 192 tokens, nano, and micro. Right: loss per unit and perplexity.

The fair unit

On this passage nano loses 1.495 nats per character and micro 3.938 nats per token — 2.6 times as much per unit, which says nothing by itself, since the passage has 0.28 of micro’s tokens per byte against 0.99 of nano’s characters per byte. (nano reads 64 characters at most, so it scores the passage in 60-character pieces.) In perplexity, micro is as uncertain as a choice among 51 tokens, nano among 4.5 characters. Per byte the comparison is fair: micro needs 1.577 bits per byte and nano 2.098. On the other two passages micro scores 1.630 and 1.482, nano 2.473 and 1.916.

Both sit far below the baselines: 8 bits per byte for guessing bytes uniformly, 3.64 for guessing micro’s tokens uniformly. For scale, GPT-2’s largest model reached 0.93 bits per byte on Wikipedia text (enwik8) in 2019 without training on it. Bits per byte also links loss to compression: a model at 1 bit per byte could, with arithmetic coding, store English text in one-eighth of its UTF-8 size — which is why some researchers describe pretraining as learning to compress the corpus.

Perplexity is still the headline metric in many papers, but it is only comparable between models that share a tokeniser and a test set. Across tokenisers, use bits per byte or bits per character.

Terms

Perplexity: e raised to the loss in nats: the effective number of equally likely choices.
Bits per byte: Total code length in bits divided by the text’s length in bytes; independent of tokenisation.
Nat: The unit of information with natural logarithms; 1 nat ≈ 1.443 bits.
Baseline: A trivial predictor, such as a uniform guess, that a model must beat.

Losses per token depend on the tokeniser; bits per byte do not — use them to compare models.

Sources

Radford et al., “Language Models are Unsupervised Multitask Learners”, OpenAI, 2019, Table 3 (enwik8 bits per byte).