Drills: Data.
Five questions at a time from the whole volume, until the path from a crawl to a training mix is familiar.
This plate draws rounds of five from the questions and predictions of the eleven plates before it, with the reasoning and source plate after each answer.
One idea runs through the volume: every stage of a data pipeline is a decision with a cost on each side — what is kept and what is lost, what is caught and what slips through, which language or register or source is favoured — and the model inherits every one of those decisions.
Below: a round of five. Answer each, read the explanation, and press “Next round” for another five.
- Trace a crawl through a data pipeline.
- Compute Jaccard similarity and LSH thresholds.
- Explain the biases of quality and safety filters.
- Reason about mixtures, synthetic data and contamination.
What to carry forward
A crawl is mostly not training text (plate IX.1). Extraction removes the boilerplate that would otherwise repeat millions of times (plate IX.2); language identification sorts by language and fails on short or under-represented text (plate IX.3); deduplication with MinHash and LSH finds near-copies without comparing every pair (plate IX.4). Quality filters define quality — perplexity keeps repetition and drops other registers (plate IX.5) — and safety filters trade privacy and harm against false positives that fall unevenly (plate IX.6).
A training mix adds code, mathematics and books on purpose (plate IX.7) and sets shares under a fixed budget, where every share is taken from another (plate IX.8). Synthetic data helps when verified and harms when recycled (plate IX.9); contamination turns tests into memory checks (plate IX.10); and the right to use text at all is being settled in audits, courts and law (plate IX.11). Vol. X turns to how much data, and how large a model, a given budget should buy.
Terms
- Pipeline
- Extraction, identification, deduplication and filtering.
- MinHash
- A signature estimating Jaccard similarity.
- Temperature sampling
- Weights ∝ size^τ.
- Contamination
- Test items in the training data.