WORLDTECHGoogle Plans Support for Shared Robots.txt AI Training Control
What happened
Google announced on October 30, 2026 that it plans to support a shared AI training control in robots.txt, giving website owners a single way to express training preferences across multiple AI providers. The commitment arrived alongside same-day announcements from Microsoft and OpenAI.
The control is based on work from the AI Preferences working group at the Internet Engineering Task Force (IETF) and builds on Google-Extended, the AI training control Google says it has offered since 2023. When a website operates with the token to opt out of training and has not explicitly specified Google-Extended, Google plans to interpret that as if the publisher had disallowed Google-Extended.
According to the post, existing company-specific controls will continue to function alongside the joint signal, so website owners can express a broad choice through the industry signal while still tailoring instructions to individual developers through controls such as Google-Extended. Websites that specifically allow Google-Extended are eligible to appear in responses and receive clicks through grounding in Gemini Apps.
How Google Will Apply the Control The plan was laid out in a Google Search Central Blog post by John Mueller, a Google Search Advocate. Instead of managing separate instructions for every AI developer, website owners will be able to use a shared policy token within robots.txt to set broad preferences across participating AI providers.
Other AI providers can handle the signal similarly, so owners can opt out across all participating providers at once or pick and choose. Google said the proposal reflects feedback from publishers, civil society groups, regulators, and other stakeholders that have called for industry-wide controls on AI training, and that support across major AI providers would produce a durable standard that works across the industry.