Google, Anthropic and others to support new AI crawling opt-out standard

What happened

Google LLC, Microsoft Corp., Anthropic PBC (a company legally required to pursue a stated public benefit) and OpenAI Group PBC today announced plans to support an upcoming website crawling standard. UPDATED 22:00 EDT / OCTOBER 30 2026 AI Google, Anthropic and others to support new AI crawling opt-out standard by Maria Deutscher Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC today announced plans to support an upcoming website crawling standard. Google is an Internet and software company based in Mountain View, and its products and services include Google Search. Anthropic is an artificial intelligence company based in San Francisco.

Many websites contain a text file titled “robots.txt” that informs crawlers what content they may access. A publisher can use the file to block scraping across the board or only make specific webpages inaccessible.

For example, a company can ask crawlers not to visit its website too often. Website operators historically used robots.txt files to manage how search engines interact with their properties.

Nowadays, the technology also helps regulate the activity of the scrapers with which AI providers collect training data (the material a model is built from). A company that wishes to disallow training-related scraping must make multiple edits to its website’s robots.txt file.

In particular, it needs to add a separate entry for every single AI provider. If an AI provider operates with multiple scrapers to collect data, each scraper requires its own robots.txt entry.

The crawling standard Google, Microsoft, Anthropic and OpenAI are backing aims to simplify the process. It would allow website operators to block all scrapers that collect training data for generative AI models with a single robots.txt entry.