WORLDTECHOpenAI to Honor Shared Robots.txt Signal for AI Training Preferences
What happened
OpenAI announced on October 30, 2026 that it will honor a shared signal in robots.txt, the file websites use to express preferences about how their content is used, so that site operators can communicate a single AI training preference to multiple participating providers at once. OpenAI said it aims to implement the signal in the coming months and will share another update with implementation details when it does.
A Voluntary Signal Shared Across Providers In its announcement, OpenAI described the effort as voluntary and said it addresses an existing burden: until now, website operators had to identify individual crawlers and maintain separate rules for each and every provider. A shared signal, the company said, lets publishers and website operators communicate the same preference to multiple participating providers at once.
The company said the new signal will not replace the controls it already provides through robots.txt, which it said will continue to apply and be respected. OpenAI framed the move as drawing on the ongoing work of the Internet Engineering Task Force’s AI Preferences working group, said it is joining other working group participants in making the announcement, and encouraged other AI providers to honor the signal.
The company described the step as moving toward an open web where people can publish and discover information and website owners have clearer, simpler ways to communicate their choices about how their content is used, and said it will continue to adapt its practices as those standards conversations evolve. How the Signal Works The signal is defined in an individual Internet-Draft titled Machine-Readable Training Preference Signal (draft-kelley-aipref-training-preference-signal-00), authored by Kevin Kelley of Anthropic, dated October 30, 2026 and expiring May 3, 2027. The IETF’s datatracker notes that an individual Internet-Draft is not endorsed by the IETF and has no formal standing in the IETF standards process.