Vol. IX · Data Plate 8 of 12data mixture, mixing weights, temperature sampling, multilinguality, trade-off, small-scale proxiesLD

Data Mixtures.

Give Chinese more of a fixed budget and Chinese improves — at English’s expense — and small runs can tell you the rate.

A training run has a token budget, and the mixture decides how it is spent. More of one source improves the model on that source and, past some point, costs it on the others; how steep each side of that trade-off is cannot be read from first principles, so labs train many small models on different mixtures and extrapolate. Data-mixing laws fitted on such proxies are now part of planning a large run.

For many sources at once, the common rule is temperature sampling: give each source a weight proportional to its size raised to a power τ between 0 and 1. With τ = 1 the mixture follows the data; smaller τ flattens it, upsampling small sources such as low-resource languages; XLM-R used τ = 0.3. The workshop shows recorded runs of micro’s shape trained on 6 million tokens split between English and Chinese in four proportions, and the arithmetic of temperature sampling.

Below: left, English and Chinese held-out loss of four recorded runs that differ only in their share of Chinese. Right, temperature sampling over four sources each a tenth the size of the last: their shares of the mix, and how many times each is repeated.

What you will learn
Workshop
Left: recorded runs, micro’s shape, 3 000 steps on a 6-million-token budget with 10%, 30%, 50% or 70% Chinese; English (blue) and Chinese (red) held-out loss. Right: four sources of sizes 1 000, 100, 10 and 1 (in any unit), sampled with temperature τ for a budget equal to their total: share of the mix (bars) and epochs of each source (numbers).

Every share is taken from another

The four recorded runs differ only in how their 6 million tokens are split. Moving from 10% to 70% Chinese makes English held-out loss 0.208 nats worse and Chinese 0.505 nats better: English goes from 4.100 to 4.308, Chinese from 5.181 to 4.677. The curves are not mirror images, because the two languages share a vocabulary, a network and some content (the course corpus pairs each English plate with its Chinese translation), so some of what is learned in one transfers. The course’s full training corpus is 47% Chinese by tokens, close to the middle of this range.

With many sources the same arithmetic runs through temperature sampling. At τ = 1 the smallest of four sources gets 0.1% of the mix; at τ = 0.3 it gets 6.7% and is read 74.5 times while the largest is read 0.59 times; at τ = 0 all four get a quarter. Upsampling buys coverage of rare languages and domains at the price of repetition, and repetition has a ceiling (plate X.6). Because these trade-offs depend on the sources, teams fit them on small proxy runs like these before committing a large budget.

Mixtures need not be fixed. Many runs change the mix during training — more high-quality and specialised data near the end — which is the subject of mid-training (Vol. XII).

Terms

Mixture weights
The share of the training budget given to each source.
Temperature sampling
Weights proportional to size^τ; τ < 1 upsamples small sources.
Proxy run
A small training run used to predict the effect of a choice on a large one.
Transfer
Learning on one source that improves performance on another.
Under a fixed budget, every share is taken from another source; temperature sampling sets the shares, and small proxy runs measure the trade-off.

Sources

  1. Conneau et al., “Unsupervised Cross-lingual Representation Learning at Scale”, ACL 2020 (XLM-R, α = 0.3).
  2. Ye et al., “Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance”, arXiv:2403.16952, 2024.