WORLDTECHGoogle, OpenAI, Anthropic and Microsoft back new robots.txt control against AI scraping
Google, OpenAI, Anthropic and Microsoft have committed to honoring a new robots.txt directive that lets website publishers state whether their content may be scraped for generative AI training, according to Engadget.
A shared signal instead of crawler-by-crawler rules
According to the report, the four companies will support a common signal in robots.txt that blocks scraping for AI training. Engadget notes that website owners currently have to identify individual crawlers and write separate rules for each provider, a task the new approach is meant to simplify.
The signal rests on a proposal under consideration at IETF, the open standards organization for the internet. The companies have decided to adopt it; it was written by Anthropic technical lead for standards Kevin Kelley.
How publishers would configure it
Publishers would add a dedicated user agent for AI training to their robots.txt file. Engadget gives the example of a rule named __usage_ai-training with a Disallow entry for the whole site, which would block crawlers gathering data for AI training.
An Allow line can carve out exceptions. The report cites a press kit as an example of a section a site could leave open to AI training while the rest of the site stays blocked.
Open questions
- When the signal will take effect across the four companies' crawlers?
- Whether the IETF proposal will be formally adopted as a standard?