Updated: 3 November 2026
Until last week, a website owner who did not want their content used to train AI models had to write separate rules for every company: GPTBot for OpenAI, Google-Extended for Google, ClaudeBot for Anthropic, and so on. Miss one, and your blog posts, product descriptions and pricing pages could end up inside someone's next model.
On 30 October 2026, Google, Microsoft, OpenAI and Anthropic announced support for one shared robots.txt signal for AI training preferences, based on a draft at the IETF (the body that writes internet standards). One rule in one file, honoured by four of the biggest AI companies.
This guide explains, in plain language, what the new rule is, what it blocks and what it does not, whether blocking AI training hurts your Google ranking, and how to decide what is right for your business website. It includes copy-paste robots.txt examples.
Contents
- Quick answer
- What was announced
- The new robots.txt rule (copy-paste)
- What it covers, and what it does not
- Will blocking AI training hurt my Google ranking?
- When each company starts honouring it
- Should your business block AI training? A decision guide
- 5 ready-made robots.txt setups
- What robots.txt cannot do
- Frequently asked questions
Quick answer
On 30 October 2026, Google, Microsoft, OpenAI and Anthropic agreed to support a shared robots.txt signal for AI training. You add a group with User-Agent: __usage_ai-training and Disallow: / (with optional Allow lines) to tell these companies not to train AI models on your content. It controls training only: it does not stop crawling, does not affect Google Search rankings, and does not remove you from AI Overviews or AI answers, which have separate controls. Anthropic already supports it, Microsoft from 1 February 2027, Google in early 2027 and OpenAI "in the coming months". Keep your existing GPTBot, Google-Extended and ClaudeBot rules for now. Most businesses should allow AI search and answers (for visibility) and decide on training based on how valuable and original their content is.
What was announced
- Date: 30 October 2026, with posts from Google, Microsoft and OpenAI, and an Anthropic message to the IETF AI Preferences (AIPREF) working group.
- The draft: draft-kelley-aipref-training-preference-signal-00, an individual Internet-Draft. It is not yet an official IETF standard, and the definition of "AI training" comes from the AIPREF working group's vocabulary draft.
- What "AI training" means here: "Using an asset to modify the learned parameters of a generative AI model." In simple words: teaching a chatbot or image model using your content.
- Enforcement: none. Like all robots.txt rules, it is voluntary. OpenAI calls its participation a "voluntary effort".
- Who has not committed yet: Apple, Meta and Amazon were not named as supporting the shared token.
The new robots.txt rule (copy-paste)
The signal uses a special "policy token" in the User-Agent line instead of a crawler name. It uses normal robots.txt path matching; no new directives are introduced.
Block AI training on the whole site:
User-Agent: __usage_ai-training
Disallow: /Block AI training, but allow one folder (for example, your press kit):
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Important details:
- A rule under
User-agent: *does not count as an AI training preference. You must use the token by name. - Company-specific tokens like
Google-Extendeddo not count as a shared opt-out for other companies. Keep them alongside the new rule. - If you have both, the more specific company token wins for that company. For example, Google says
Google-Extendedtakes precedence over the shared token.
What it covers, and what it does not
| Use of your content | Controlled by the new token? | What controls it instead |
|---|---|---|
| Training generative AI models (ChatGPT, Gemini, Claude, Copilot models) | ✅ Yes | The new token, plus existing company tokens |
| Crawling (bots visiting your site) | ❌ No | Normal robots.txt rules for each bot |
| Google Search ranking and indexing | ❌ No | Googlebot rules; Google says Google-Extended is not a ranking signal |
| Google AI Overviews and AI Mode | ❌ No | Googlebot rules and Search Console settings |
| "Grounding": AI tools reading your page live to answer a question | ❌ No (it does not change model parameters) | Company-specific controls; Microsoft has a separate AI grounding setting; Google-Extended also covers Gemini grounding |
| Bing search results | ❌ No | Microsoft says preferences do not change eligibility for normal Bing results |
| Non-generative models (ranking, classification, spam filters) | ❌ No | Not covered by the definition |
Will blocking AI training hurt my Google ranking?
No, according to Google and Microsoft. Google says Google-Extended (which the shared token maps to) does not affect inclusion in Google Search and is not a ranking signal. Microsoft says the preferences do not change eligibility for traditional Bing results; in fact, the new token gives Bing site owners an option they did not have before, because previously opting out of Bing training was tied to Bing visibility.
The real trade-off is AI visibility, not search ranking. More and more customers ask ChatGPT, Gemini, Copilot and Google's AI answers instead of clicking ten blue links. Those answers come partly from what models learned in training and partly from live "grounding" on web pages. Blocking training can reduce how well a model "knows" your brand over time; blocking grounding or AI search crawlers can remove you from live answers. For more on this, read our guide to SEO in the age of AI search.
When each company starts honouring it
| Company | Status | Notes |
|---|---|---|
| Anthropic (Claude) | Already started implementing | Complements ClaudeBot, Claude-User and Claude-SearchBot controls |
| Microsoft (Bing, Copilot) | From 1 February 2027 | Bing Webmaster Tools transition settings open 15 December 2026; old NoArchive/NoCache tags stop counting as AI controls after 1 March 2027 |
| Google (Gemini) | "Early 2027" | Google-Extended stays and takes precedence |
| OpenAI (ChatGPT) | "In the coming months" | GPTBot rules continue to work |
| Apple, Meta, Amazon | Not committed | Use their own tokens (for example Applebot-Extended) |
Should your business block AI training? A decision guide
| Your website is mainly… | Our usual recommendation |
|---|---|
| A local or service business (clinic, shop, contractor, agency, manufacturer) | Allow training and AI search. You want AI tools to know who you are, where you are and what you do. Your value is the service, not the web text. |
| A B2B or SaaS company | Allow on marketing pages, block on docs, pricing calculators or gated resources if they are a competitive asset. |
| A publisher, educator or course creator | Consider blocking training on paid or original content; keep a public summary area allowed so AI tools can still cite you. |
| A photographer, designer or artist portfolio | Block training on your work folders; allow the about and contact pages. |
| E-commerce | Allow product and category pages (AI shopping answers are a growing channel); block anything with customer data, which should never be public anyway. |
Whatever you choose, do not block Googlebot or Bingbot. That removes you from search entirely, and it is the most common robots.txt mistake we see. If your site is not showing on Google, check this first; our guide on why your website is not showing on Google covers the other causes.
5 ready-made robots.txt setups
1. Local business: allow everything (most common)
User-agent: *
Allow: /
Sitemap: https://www.yourdomain.com/sitemap.xml2. Block AI training everywhere, keep search and AI answers
User-Agent: __usage_ai-training
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.yourdomain.com/sitemap.xmlNote: Google-Extended also covers grounding in Gemini apps, so blocking it may reduce how Gemini uses your pages in answers. Decide accordingly.
3. Block training on premium content only
User-Agent: __usage_ai-training
Disallow: /courses/
Disallow: /reports/
Disallow: /members/
User-agent: *
Allow: /
Sitemap: https://www.yourdomain.com/sitemap.xml4. Portfolio: protect the work, allow the business pages
User-Agent: __usage_ai-training
Disallow: /portfolio/
Disallow: /gallery/
Allow: /about/
Allow: /contact/
User-agent: *
Allow: /5. Never do this (it blocks Google Search)
User-agent: *
Disallow: /What robots.txt cannot do
- It is a request, not a lock. Bots that ignore robots.txt need network-level bot management (for example, firewall or CDN bot rules).
- It does not remove content already used. Preferences apply to content retrieved under the robots.txt in effect at that time.
- It does not protect private data. Anything sensitive should be behind login, never just "hidden" by robots.txt, which is itself public. Personal data on public pages is also a DPDP risk; see our DPDP compliance checklist.
- The standard may change. It is still a draft, and other approaches exist (an HTTP
Content-Usageheader, Cloudflare's Content Signals, the RSL licensing standard). Review your settings every six months.
How Vegavaan can help
Vegavaan Info Solutions builds and maintains SEO-ready business websites. We can audit your robots.txt, sitemap and AI crawler settings, set up the right mix of search visibility, AI-answer visibility and training protection for your business, and add bot protection where needed, without breaking your Google rankings.
👉 Get a free robots.txt and AI visibility check or see our website design and development services.
Frequently asked questions
How do I stop AI companies from training on my website?
Add a robots.txt group with User-Agent: __usage_ai-training and Disallow: /. Google, Microsoft, OpenAI and Anthropic announced support for this shared signal on 30 October 2026. Keep company-specific rules such as GPTBot, Google-Extended and ClaudeBot until every company has implemented it.
Does blocking AI training affect my Google ranking?
Google says Google-Extended, which the shared token maps to, does not affect Google Search inclusion and is not a ranking signal. Microsoft says the preferences do not affect normal Bing results.
Will blocking AI training remove me from Google AI Overviews?
No. AI Overviews and AI Mode are controlled by Googlebot rules and Search Console settings, not by the shared AI training token.
What is the difference between AI training and AI grounding?
Training uses your content to change a model's learned parameters. Grounding is when an AI tool reads your page live to answer a question. The new shared token covers training only.
When will Google and OpenAI honour the new robots.txt token?
Google says early 2027, Microsoft from 1 February 2027, OpenAI "in the coming months", and Anthropic has already started implementing it.
Should a small business block AI training?
Usually not. Most local and service businesses benefit when AI tools know their name, location and services. Blocking makes more sense for publishers, course creators, artists and companies with valuable original content.
Conclusion
For the first time, website owners have one simple, shared way to say "do not train AI on my content" to four of the biggest AI companies. It is voluntary, it is still a draft, and it covers training only, not search or AI answers. For most businesses the right move is to stay visible in search and AI answers, protect only what is truly valuable, and review the settings as the standard matures.
Not sure what your robots.txt says right now? Talk to Vegavaan.
Sources: PPC Land: Google, Microsoft, OpenAI and Anthropic back one AI training opt-out; Coywolf: Shared robots.txt AI training signal.
