OpenAI、Google、Microsoft、Anthropic 支持统一的 AI 抓取退出标准
The biggest names in AI are backing a new way to opt out of scraping
- 1OpenAI、Google、Microsoft、Anthropic 四家宣布将采用统一标准，网站可一次声明拒绝多家 AI 抓取
- 2统一 policy token 信号由 Anthropic 标准技术负责人 Kevin Kelley 在 IETF 提出，写入网站 robots.txt 文件
- 3落地节奏不一，Anthropic 已采用该信号，Google 与 Microsoft 计划明年支持，OpenAI 称数月内落实
- 4Microsoft 额外提供更细控制，可分别退出 AI 训练或 AI grounding，网站无论偏好如何仍会出现在 Bing 搜索结果
AI 抓取治理正从逐家沟通转向统一协议，四大厂商共同背书让网站表达训练偏好的成本显著下降。
- 影响：内容方一次 robots.txt 配置即可覆盖多家 AI 公司，数据授权管理成本随之下降；依赖公开网页做训练或 grounding 的团队要为更多站点退出做准备；搜索收录不受该信号影响，Bing 流量逻辑保持不变。
- 参考：可考虑盘点自有站点的 robots.txt 与现有抓取协议，评估是否启用统一退出信号；内容授权方值得把 RSL 收费、Cloudflare 默认拦截与该标准组合比较；采购 AI 服务时可核实厂商对退出信号的执行承诺。
- 依据：四家厂商的采用表态为官方口径，暂无独立验证；标准由 Anthropic 员工在 IETF 提出，各家落地时间表由厂商自行公布。
Emma Roth
is a news writer who covers the streaming wars, consumer tech, crypto, social media, and much more. Previously, she was a writer and editor at MUO.
OpenAI, Google, Microsoft, and Anthropic say they will adopt a new standard that publishers can use to opt out of being scraped by AI, as reported earlier by Search Engine Roundtable. With this control, publishers can apply their AI training preferences across multiple AI companies, instead of setting them individually.
The shared “policy token” signal is proposed by Kevin Kelley, a technical lead for standards at Anthropic, as part of the Internet Engineering Task Force (IETF). It will live within a website’s robots.txt file, which has long signaled which web crawlers can and can’t scrape a site — whether for search indexing, archiving, or AI training. Currently, sites have to signal their AI training preferences separately to each AI company.
The announcement comes as publishers grapple with the continuing influx of AI crawlers scraping their sites and feeding their content into AI models. Some companies and organizations have attempted to combat unauthorized scraping, with the Really Simple Licensing (RSL) standard giving publishers the ability to charge crawlers that visit their sites, and Cloudflare blocking AI crawlers by default.
OpenAI, Google, Microsoft, and Anthropic say they will restrict their AI crawlers from accessing a site if they encounter the shared token. Microsoft, however, will offer more granular controls that will give publishers the ability to opt out of AI training or AI grounding, the process of pointing an AI model to a verified source of information to reduce hallucinations. Microsoft notes that websites will still appear in Bing’s search results regardless of their preferences.
While Anthropic has already adopted the signal, Google and Microsoft plan to support it next year. OpenAI says it will implement it in the “coming months.”
Follow topics and authors from this story to see more like this in your personalized homepage feed and to receive email updates.
- Emma Roth
来源：The Verge · AI · theverge.com