Vol. IX · Data Plate 2 of 12HTML extraction, boilerplate removal, text density, link density, trafilatura, WARC, WETDE

From HTML to Text.

Strip the tags and you keep the menus; keep only dense blocks of prose and you get the article.

Crawls store the raw HTML of every page, and Common Crawl also publishes a plain-text version made by simply removing the tags. Removing tags is not the same as finding the text: what remains still includes every menu item, “Log in”, the cookie banner, the copyright line and the newsletter box — identical across thousands of pages from the same site.

Better extractors look at the structure. Blocks inside nav, footer or script elements go; so do blocks that are mostly links or too short to be prose. The FineWeb team found that extracting from the raw HTML with a dedicated tool (trafilatura) gave measurably better models than using the tag-stripped text. The workshop compares the two approaches on one simulated page.

Below: one page of the simulated crawl: its HTML, the text from stripping tags, and the text a density-based extractor keeps, with the blocks it dropped.

What you will learn
Workshop
One simulated page. Left: the HTML. Middle: the text left by stripping all tags. Right: the text kept by a density-based extractor, with the blocks it dropped listed below it.

The menus are the problem

On the first page of the simulated crawl, an English article, stripping tags leaves 209 words and density-based extraction keeps 180, dropping 4 blocks: a script, the navigation bar, the newsletter box and the copyright footer. The difference looks small on one page. Across a crawl it is not, because boilerplate is the same everywhere: a site with a million pages contributes its menu a million times, and those repeated strings are exactly what a model learns most easily and memorises (plate VIII.7).

Extraction also loses things. Tables, code blocks, mathematical notation and short list items can fall below a density threshold, and a forum page whose real content is a series of short replies may be stripped almost bare. FineWeb’s comparison found that a careful extractor working from the raw HTML produced better models than the tag-stripped text despite keeping fewer tokens — quality of text, not quantity, decided the result.

The raw HTML archives (WARC files) are several times larger than the plain-text versions (WET files); working from the HTML costs more storage and processing, which is why many early datasets used the cheaper text.

Terms

Boilerplate
Text repeated across pages of a site: menus, footers, banners, notices.
Extraction
Turning a page’s HTML into its main text.
Link density
Links per word in a block; high values mark navigation.
WARC and WET
Common Crawl’s raw HTML archives and their tag-stripped text versions.
Stripping tags keeps the menus and footers; extraction keeps the prose — and the model learns from whichever it is given.

Sources

  1. Penedo et al., “The FineWeb Datasets”, arXiv:2406.17557, 2024, section 3 (extraction from WARC with trafilatura against the WET files).
  2. Barbaresi, “Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction”, ACL 2021 demonstrations.