23 Daily — According to theverge.com, on October 30, 2026, major AI companies including OpenAI, Google, Microsoft, and Anthropic jointly announced the adoption of a new standard allowing websites to opt out of AI training data scraping uniformly via the robots.txt file. This move addresses concerns from site operators about excessive AI data scraping and offers a simplified way to manage consent across multiple AI providers.
Introducing a Shared "Policy Token" in robots.txt
The "policy token" signal, proposed by Kevin Kelley, a technical standards lead at Anthropic within the Internet Engineering Task Force (IETF), will be embedded in websites' robots.txt files. Historically, robots.txt directs web crawlers on which pages can or cannot be scraped for search indexing or archiving purposes. Now, it will also control access by AI training crawlers. Previously, sites had to set these preferences individually for each AI company; the new standard allows a single, unified setting, improving management efficiency.
Industry Adoption and Implementation Timeline
Anthropic has already adopted the shared token, with Google and Microsoft planning to support it next year. OpenAI aims to implement it within the coming months. Microsoft’s approach offers more granular control, allowing publishers to opt out of AI training or AI grounding—where AI models reference verified sources to reduce hallucinations. Microsoft clarified that websites will still appear in Bing search results regardless of these preferences.
Editor's take: Implications and Validation for Entrepreneurs
For entrepreneurs relying on website traffic and content licensing, this standard provides a clearer, more controllable way to manage how their content is used by major AI models, potentially impacting licensing business models and brand protection. Compared to previous fragmented controls, it significantly lowers management costs and legal risks. Entrepreneurial teams can verify the standard’s effectiveness by monitoring changes to their robots.txt settings alongside web traffic and AI crawler activity, assessing how the new framework affects their operations.