Vol. IX · Data Plate 7 of 12code data, mathematics data, books, papers, Wikipedia, data mix, Llama 3, compressionDE

Beyond the Web.

Add code, mathematics and books on purpose — and notice that each costs a different number of tokens per byte.

Large training sets combine sources. Code from public repositories teaches syntax, exact structure and long-range consistency; mathematical text teaches symbolic manipulation; papers and books supply long, coherent argument; encyclopaedias supply dense facts. Llama 3’s final pretraining mix, by its report, was roughly 50% general knowledge, 25% mathematics and reasoning, 17% code and 8% multilingual text — far more code and mathematics than the web contains on its own.

Sources differ in how they tokenise and how predictable they are, which matters for both cost and evaluation. A tokeniser trained mostly on prose compresses code poorly: indentation, punctuation and identifiers split into many short tokens. The workshop measures the course tokeniser and micro on four kinds of text — English prose, Chinese prose, Python code and a line of mathematics — and shows Llama 3’s published mix beside them.

Below: left, bytes per token of the course tokeniser and bits per byte under micro, for four kinds of text. Right, the Llama 3 pretraining mix.

What you will learn
Workshop
Left: four texts — an English and a Chinese passage from the held-out course text, a short Python program and a line of mathematics — with bytes per token under the course tokeniser (blue) and bits per byte under micro (red). Right: Llama 3’s final pretraining mix by token share (arXiv:2407.21783, section 3.1.2).

Different sources, different costs

The course tokeniser, trained on prose, packs 3.29 bytes into an average English token and 4.13 into a Chinese one, but only 1.74 into a token of Python: spaces, brackets and short identifiers each take a token of their own. micro, which saw almost no code, needs 4.16 bits per byte on it against 1.80 on English. A production tokeniser trained on a mix that includes code does far better, which is one reason vocabularies have grown and code-heavy models use their own merges for runs of spaces.

Llama 3’s report puts its final mix at about half general knowledge and a quarter mathematics and reasoning, with 17% code — shares chosen by experiment, since the web alone holds far less of either. The case for code is partly direct (models are asked to write it) and partly indirect: Aryabumi and colleagues found that including code in pretraining improved performance on non-code reasoning and knowledge tasks too. Books and papers are valuable for long, coherent text, but they raise the questions of licence and consent that close this volume (plate IX.11).

Upsampling a small, high-value source means repeating it: a source that is 1% of the available text but 10% of the mix is read about ten times as often as the rest, which brings the memorisation risks of plate VIII.7 back in.

Terms

Data source
A collection of one kind of text: web, code, papers, books, encyclopaedia.
Upsampling
Giving a source a larger share of training than its share of the available text.
Data mix
The proportions in which sources are sampled during training.
Bytes per token
How much text one token covers; lower means a source costs more tokens.
Code, mathematics and books are added and upsampled on purpose; they teach what the web teaches poorly and cost different numbers of tokens per byte.

Sources

  1. Llama Team, AI @ Meta, “The Llama 3 Herd of Models”, arXiv:2407.21783, 2024, section 3.1.2.
  2. Kocetkov et al., “The Stack: 3 TB of permissively licensed source code”, arXiv:2211.15533, 2022.
  3. Aryabumi et al., “To Code, or Not To Code? Exploring Impact of Code in Pre-training”, arXiv:2408.10914, 2024.