Data, Consent and Law.
The web’s signals of consent were written for search engines; their meaning for training is being worked out in audits, courts and law.
Most web text is protected by copyright, but crawling for search has long relied on a voluntary convention: a site’s robots.txt file says which crawlers may visit, and well-behaved crawlers obey. When AI developers began publishing the names of their crawlers — OpenAI documented GPTBot in August 2023 — sites could opt out of training specifically, and many did. An audit by Longpre and colleagues found that between April 2023 and April 2024 about 5% of all tokens in three major open datasets, and about a quarter of tokens from the most actively maintained sources, became restricted by robots.txt; terms of service restricted far more.
Courts and legislators have begun to act. In June 2025 a US district court held in Bartz v. Anthropic that training on lawfully acquired books could be fair use, but that obtaining pirated copies was not protected; Anthropic then agreed a settlement of $1.5 billion. Other cases, including The New York Times’s suit against OpenAI and Microsoft filed in December 2023, continue. In the EU, the AI Act entered into force in August 2024, and from 2 August 2025 providers of general-purpose models placed on the market must publish a summary of their training content using a template set by the Commission. This plate reports these as events, not legal advice; the law differs by country and is still changing.
Below: left, a timeline of the protocols, documentation practices, cases and laws that shape training data. Right, the measured tightening of consent signals, applied to a 15-trillion-token dataset.
- Describe robots.txt and why it matters for training.
- Summarise the measured decline in openly usable web data.
- Report the main legal developments without overstating them.
- Explain the role of dataset documentation.
Consent signals, courts and statutes
The audit’s numbers describe a fast change. In one year about 5% of all tokens in C4, RefinedWeb and Dolma came under robots.txt restrictions — for a dataset the size of FineWeb, 750 billion tokens — and about 25% of the tokens from the most critical, actively maintained sources such as news and reference sites; terms of service restricted 45% of C4. robots.txt was designed for search crawlers and is voluntary, so these signals bind only crawlers that honour them, and they apply going forward, not to text already collected. Their effect is to make the newest high-quality text the least available to open datasets.
The legal picture is unsettled and differs between countries. In the US, the Bartz ruling drew a line between how material was acquired and how it was used, and other cases may draw others. In the EU, the AI Act’s transparency duty asks for a summary rather than a list of works, alongside obligations about respecting rights holders’ opt-outs. Documentation practices such as datasheets for datasets — what was collected, from where, how it was filtered, what it contains — predate the laws and remain the practical basis for answering such questions. Engineers working on data should treat licences and opt-outs as constraints on design, and take legal advice for their own jurisdiction.
Terms
- robots.txt
- A file in which a site states which crawlers may visit which pages; honoured voluntarily.
- Opt-out
- A rights holder’s signal that their content should not be used, for example by blocking a named AI crawler.
- Fair use
- A US legal doctrine permitting some uses of copyrighted work without permission, decided case by case.
- Datasheet
- Documentation of a dataset’s origin, composition, collection and intended uses.
Sources
- Koster, “A Standard for Robot Exclusion”, 1994; standardised as RFC 9309, 2022.
- Gebru et al., “Datasheets for Datasets”, Communications of the ACM 64(12), 2021 (arXiv 2018).
- Longpre et al., “Consent in Crisis: The Rapid Decline of the AI Data Commons”, NeurIPS 2024 Datasets and Benchmarks, arXiv:2407.14933.
- Bartz v. Anthropic PBC, N.D. Cal., order on fair use, June 2025; class settlement announced 5 September 2025.
- The New York Times Co. v. Microsoft Corp. and OpenAI, S.D.N.Y., complaint filed December 2023.
- Regulation (EU) 2024/1689 (AI Act), in force 1 August 2024; obligations for general-purpose AI models from 2 August 2025; European Commission template for the public summary of training content, July 2025.