Vol. IX · Data Plate 1 of 12Common Crawl, web crawl, data pipeline, filtering, FineWeb, pretraining dataDE

The Web as Corpus.

Billions of pages go in; a small, cleaned fraction comes out — and every step of the funnel is a decision.

A web crawler follows links and saves what it finds. Common Crawl, a non-profit archive running since 2007, publishes a new crawl most months; its March 2025 crawl alone held 2.74 billion pages, 455 TiB of uncompressed content from 46.7 million hosts. Most open pretraining sets begin there: FineWeb (2024) distilled 15 trillion tokens from 96 Common Crawl snapshots.

What a crawl contains is not what a model should read. There are navigation menus, cookie notices and footers around every article; pages in hundreds of languages; the same text mirrored thousands of times; lists of keywords written for search engines; and pages with no prose at all. A data pipeline removes these step by step. The workshop runs a miniature version on a simulated crawl of pages whose kind is known, so every stage can be audited.

Below: left, how many pages of a simulated 120-page crawl survive each stage of the pipeline, coloured by what each page really is. Right, the scale of one real Common Crawl snapshot.

What you will learn
Workshop
Left: a simulated crawl of 120 pages built from the course corpus with known kinds — English articles, Chinese articles, other languages, exact copies, near copies, boilerplate-only pages, keyword spam and code — passed through text extraction, English identification, exact and near deduplication and quality rules; stacked bars show the kinds remaining after each stage. Right: Common Crawl’s March 2025 snapshot.

A funnel of decisions

In this simulated crawl of 120 pages, 40 survive the pipeline, a yield of 33%. Extraction discards the pages that are all menus and links; English identification removes the Chinese, French, German and Spanish pages, the code and the keyword spam; exact deduplication catches nothing, because each mirror carries its own page title, and near deduplication removes every mirror and all but 1 of the lightly edited copies. 39 of the 43 English articles survive; a few genuine articles are lost on the way and one near-copy slips through, as happens in every real pipeline.

A real crawl is twenty million times larger. Common Crawl’s March 2025 snapshot held 2.74 billion pages from 46.7 million hosts; FineWeb kept 15 trillion tokens from 96 such snapshots after stages much like these. The choices at each stage — which language, what counts as a duplicate, what counts as quality — decide what the model reads, and so what it knows and how it writes. The rest of this volume takes the stages one at a time.

The pipeline’s order matters for cost: cheap filters such as extraction and language identification run first, on everything, so that expensive ones such as model-based quality scoring run only on what remains.

Terms

Web crawl
A snapshot of pages collected by following links.
Common Crawl
A non-profit archive of monthly web crawls, the starting point of most open pretraining sets.
Data pipeline
The sequence of extraction, identification, deduplication and filtering that turns a crawl into training text.
Yield
The share of crawled material that survives the pipeline.
Pretraining text is what survives a funnel of extraction, language identification, deduplication and filtering — each a decision.

Sources

  1. Common Crawl, “March 2025 Crawl Archive Now Available”, Common Crawl blog, 2025.
  2. Penedo et al., “The FineWeb Datasets”, NeurIPS 2024 Datasets and Benchmarks, arXiv:2406.17557.