Quality Filters.
Every quality filter is a definition of quality — and each one keeps something it should not and drops something it should keep.
Three families of filters are in common use. Heuristic rules, such as those used for DeepMind’s Gopher (2021), check simple statistics: the number of words, their mean length, the share of symbols, how many lines end in punctuation, whether common words such as “the” and “and” appear. Perplexity filters, as in CCNet (2020), score each document with a small language model trained on reference text such as Wikipedia and keep what that model finds probable. Classifier filters train a model to recognise text like a chosen target: FineWeb-Edu kept 1.3 trillion of FineWeb’s 15 trillion tokens using a classifier trained on a larger model’s ratings of educational value.
Each family encodes a definition of quality, and each has blind spots. The workshop applies two of them to eight short texts: a paragraph of the course, a forum post, a recipe, a legal notice, a line of boilerplate repeated eight times, keyword spam, a short poem and a list of numbers. The perplexity filter uses micro — trained on the course corpus — as its reference model, so “quality” here means “like the course”.
Below: left, each text’s bits per byte under micro, with the texts a perplexity threshold keeps in green. Right, which of the Gopher-style rules the selected text passes.
- Name the three families of quality filters.
- Apply a perplexity filter and read what it keeps.
- Apply heuristic rules and read what they reject.
- Explain why every quality filter carries a bias.
Quality means “like the reference”
micro scores the course paragraph at 1.87 bits per byte — the kind of text micro was trained on — and the repeated line “Click here to subscribe.” at 1.83, as low or lower: once a text starts repeating, every model finds it predictable. The forum post (2.80), the poem (2.60) and the keyword spam (2.52) score worst. With the threshold at 2.3 bits per byte the filter keeps 4 of the eight texts: course prose and boilerplate in, an ordinary person’s forum post and a poem out. That is the signature of perplexity filtering — it keeps text that resembles its reference, including degenerate repetition, and drops text in other registers, however human.
The heuristic rules fail differently. Every one of these samples fails the first rule, a minimum length of 50 words; beyond that, the repeated boilerplate fails for lacking common English words, the spam for its missing punctuation, and the list of numbers almost everything. The rules are written for English prose and say nothing about Chinese, code or verse. Classifier filters such as FineWeb-Edu’s are more flexible but just as definitional: they keep text like what the classifier was taught to rate highly, 9% of FineWeb in its case. Data teams therefore check what a filter removes, not only how much, and mix filtered text with other sources rather than relying on one definition.
Terms
- Heuristic filter
- Hand-written rules on simple text statistics.
- Perplexity filter
- Keeping text that a reference language model finds probable.
- Classifier filter
- Keeping text that a trained classifier scores as resembling a chosen target.
- Register
- A variety of language suited to a context: formal prose, chat, verse, legal text.
Sources
- Rae et al., “Scaling Language Models: Methods, Analysis & Insights from Training Gopher”, arXiv:2112.11446, 2021, appendix A (MassiveText quality filters).
- Wenzek et al., “CCNet”, LREC 2020.
- Penedo et al., “The FineWeb Datasets”, arXiv:2406.17557, 2024 (FineWeb-Edu).