Google, OpenAI, Microsoft, and Anthropic backed a shared AI training opt-out signal for robots.txt on October 30 — one entry instead of a separate rule for every crawler. It's voluntary, unenforceable, and only Anthropic is honoring it today. If you've never deliberately blocked an AI crawler, this changes nothing for you. If you have, it's a five-minute fix.
What Actually Changed on October 30#
The four companies announced plans to back a shared standard, reported by Coywolf as the token __usage_ai-training. It blocks AI training scrapers across all four companies with one robots.txt entry, instead of separate Google-Extended, GPTBot, and Claudebot rules.
The rollout is staggered:
- Anthropic — honoring it already
- OpenAI — "coming months"
- Microsoft — targeting February 2027
- Google — aiming for early 2027
The four have already filed a draft with the IETF — this is headed toward an actual internet standard, not a one-off announcement.
Does the AI Training Opt-Out Affect Your Site?#
No, probably not. If you've never written a Google-Extended or GPTBot rule, the new token sits next to a decision you haven't made. It only matters if you already block AI training scrapers, or want to start.
It's also narrower than the headline suggests. The token covers training-data collection only — it says nothing about whether ChatGPT, Gemini, or Copilot can still crawl your pages at answer-time to cite you, which is the other direction of the same crawler-permissions problem. Microsoft's reported "AI Grounding" control for Copilot and 365 looks like a separate lever from this training token, and neither source covering the story fully reconciles the two.
At WebEpex we checked every live client site against this within two days of the announcement — across GCC, European, and North American accounts, plus the Indian SaaS builds we maintain — and found exactly two with a deliberate AI-crawler block already in place. Everyone else had nothing to update, which tracks with how little on-site technical blocking actually drives AI citations compared to earned coverage.
The Honest Trade-Offs#
The upside is real. One entry instead of a growing pile of crawler-specific rules is less maintenance, and a shared IETF draft is likelier to stick than a one-off blog post. OpenAI publicly pushing other providers toward the same signal adds weight, and Microsoft folding this into its own AI-search visibility tooling suggests it's thinking past just training data.
The downside: it's unenforceable. A scraper that ignores robots.txt entirely isn't stopped by this — you'd still need something like Cloudflare's bot mitigation for that. The staggered rollout also means "supported" doesn't mean "active" for three of the four companies until sometime in 2027. My read: this is a useful cleanup of a messy pile of per-crawler rules, not a new lever most businesses were missing.
How We're Handling It#
We almost skipped auditing for this one. A training-data opt-out token felt too far in the weeds to matter for most of our clients. But two accounts already block crawlers on purpose, so it wasn't nothing.
At WebEpex we added the shared token alongside their existing rules the same week it was announced, since it costs nothing to sit there inactive until Google and Microsoft catch up. We manage robots.txt directly in each client's Next.js repo rather than through a plugin, so this was a five-line pull request, not a ticket that sits for a quarter.
What Should You Actually Do This Week?#
Check whether you're currently blocking any AI crawler on purpose. If you're not, do nothing — there's no new decision here. If you are, add the shared token next to your existing rules now; it costs nothing and activates as providers roll out through 2027.
Don't confuse this with blocking AI answer engines from citing you. That's a different setting, and blocking it usually costs you visibility rather than protecting anything. If you're unsure which camp you're in, that's a two-minute check, not a redesign.
If you want someone to look at your robots.txt and tell you plainly whether any of this applies to you, send me your domain and I'll tell you straight — no pitch attached. [cal.com/webepex/growth-review]
