robots.txt for AI: GPTBot, ClaudeBot, PerplexityBot and what to block

The basics

What is robots.txt and why does it matter now with AI?

robots.txt is a plain-text file that sits at the root of your domain (yourdomain.co.uk/robots.txt). It works in groups: a User-agent line says which bot you are addressing, and beneath it go the Disallow rules (keep out of here) and Allow rules (this is fine). Since September 2022 it has been an official standard, the IETF’s RFC 9309.

Until recently, thinking about Googlebot and Bingbot was enough. Today OpenAI, Anthropic, Perplexity, Meta, Apple, Amazon and Mistral all publish their own AI crawlers, each with a different purpose. The question is no longer “do I let AI in, yes or no?” but which use of your content you accept: models being trained on it, answers citing it, or assistants reading it when a user asks. That, at heart, is what configuring a robots.txt for AI means: deciding bot by bot, not blindly.

So before you block AI bots, it pays to know what each one does. Blocking the wrong one protects nothing and can cost you visibility.

That decision has a direct impact on your AI SEO: a misplaced block can wipe you from ChatGPT Search or Perplexity without anyone telling you. If you want the full picture of how visibility is earned in search engines and answer engines, I cover it in the guide on what is GEO and how it relates to SEO and AEO.

The key concept

GPTBot, OAI-SearchBot and ChatGPT-User: why are there three bots per company?

Because each one does a different job, and blocking it has different consequences. This is the central idea behind any robots.txt for AI. OpenAI is the clearest example: there is not one ChatGPT user agent but three, and its official documentation puts it in writing: “each setting is independent of the others”.

1. Training crawlers

GPTBot, ClaudeBot, Meta-ExternalAgent, CCBot and MistralAI-Training collect text that may end up in the training data of future models. Blocking them does not remove you from any answer today: it simply signals that you do not want your future content feeding those models. What has already been collected is not deleted.

2. Search crawlers

OAI-SearchBot, Claude-SearchBot, PerplexityBot and Meta-WebIndexer build the index that the citations and links in search-backed answers come from. OpenAI is blunt about it: sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers”. These are the ones you almost never want to block.

3. Agents acting on a user’s request

ChatGPT-User, Claude-User, Perplexity-User and Meta-ExternalFetcher do not crawl on their own initiative: they open a specific URL when someone asks the assistant to read it. Here is the catch: OpenAI warns that, because these actions are initiated by a user, “robots.txt rules may not apply”, and Perplexity, Meta and Amazon say something similar. Anthropic, by contrast, states that all its bots respect robots.txt.

The special cases: Google, Apple and Microsoft

Google has no separate AI bot. AI Overviews and AI Mode are part of Search and draw on what Googlebot crawls. Google-Extended is not a crawler but a control token: it decides whether content already crawled can be used to train Gemini and for grounding, and Google says it does not affect your inclusion or ranking in Search. Apple follows the same model with Applebot and Applebot-Extended. Microsoft has no separate AI bot either: Copilot relies on the Bingbot index, and its webmaster guidelines, rewritten in February 2026 control use in Copilot through the noarchive meta tag.

Verified October 2026

Which AI crawlers read robots.txt? A verified user-agent table

This is the table I use to configure a robots.txt for AI. Every row has been checked against each company’s official documentation as of 9 October 2026; where the provider gives no detail or there is doubt, I say so. Names change often, so check the date if you land here a few months from now.

Recent changes worth keeping on your radar:

• OpenAI also documents OAI-AdsBot, which only visits pages submitted as ads in ChatGPT to check their safety. It is not a general crawler.
• Meta now separates training (Meta-ExternalAgent), Meta AI search (Meta-WebIndexer) and user requests (Meta-ExternalFetcher).
• Mistral publishes three agents: MistralAI-Training, MistralAI-Index and MistralAI-User. Its assistant is now called Vibe (formerly Le Chat).
• Google lists Google-Agent among its user-triggered fetchers; it is used by agents hosted on Google’s infrastructure to browse the web. Like the rest of that family, it “generally ignores” robots.txt rules.
• Perplexity: on 4 August 2025 Cloudflare accused it of using undeclared crawlers to evade blocks and removed it from its list of verified bots. Perplexity rejected the accusation. That is why I flag it with a caveat in the table.
• ByteDance publishes no official documentation for Bytespider that I have been able to verify. In 2024 Cloudflare identified it as the AI bot making the most requests on its network.

Tool

How do you block AI bots with a robots.txt disallow rule?

With one group per bot (or several bots in the same group) and a Disallow: / rule. The standard lets you stack several User-agent lines that share the same rules, which keeps the file shorter:

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

What many people look up as robots.txt disallow is exactly this rule: Disallow: / closes the whole site to the bots in that group. It can also be partial: Disallow: /clients/ blocks just that folder. If several rules match a URL, the most specific one wins, meaning the longest path.

To avoid mistakes, use this robots.txt for AI builder. Pick a quick preset or switch bots on and off one by one: the file is generated instantly, new lines are highlighted and the panel tells you what happens to your visibility.

# Generated at josegalan.dev/en/guides
# Check your private paths before publishing it
 
User-agent: *
Allow: /
 
Sitemap: https://www.yourdomain.co.uk/sitemap.xml

The builder deliberately leaves Googlebot, Bingbot and Applebot out of the “Block all AI” preset: blocking them also takes you out of Google, Bing and Apple’s search results. If that is genuinely what you want, you can switch them off by hand.

My recommendation

Which robots.txt for AI do I recommend for a small business?

For most small businesses, the best starting point is to let every search and user bot through and to take your time over training. If your site is mostly commercial content (services, case studies, pricing, guides), models knowing about your brand tends to work in your favour.

# Recommended robots.txt for most small businesses (October 2026)
# Goal: maximum visibility in search engines and in AI.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /my-account/
Disallow: /*?s=

# Crawler with no public official documentation
User-agent: Bytespider
Disallow: /

# Optional: if you do NOT want models trained on your content,
# remove the hashes from this block. It does not affect your
# visibility in ChatGPT Search, Perplexity or Google.
# User-agent: GPTBot
# User-agent: ClaudeBot
# User-agent: Google-Extended
# User-agent: Applebot-Extended
# User-agent: CCBot
# User-agent: Meta-ExternalAgent
# User-agent: MistralAI-Training
# Disallow: /

Sitemap: https://www.yourdomain.co.uk/sitemap.xml

Adapt the private paths to your CMS: the ones in the example are typical of WordPress with WooCommerce. The training block is commented out so that you only switch it on if it suits you. I have left out Amazonbot because it mixes product improvement and training; add it if you want to be strict.

robots.txt decides what bots can read. If you also want to suggest which pages models should read first, there is a complementary proposal: I explain what it is, who actually uses it and how much to expect from it in the guide llms.txt: what it is and how to create one.

Watch out

Is robots.txt enough? Cloudflare, CDNs and WAFs

No. You can have a perfect robots.txt for AI and still be blocking AI without realising it, or the other way round, because your CDN or firewall gets the final say before your file does.

The most important case is Cloudflare, which serves a very large share of the web. On 1 July 2025 it announced that it blocks AI crawlers accessing content without permission by default: since then, every new domain is asked whether it wants to allow them. The one-click option to block them had existed since 2024, and today it is managed bot by bot in AI Crawl Control.

Cloudflare can also serve a managed robots.txt that adds the line Content-Signal: search=yes, ai-train=no. This is its Content Signals Policy (24 September 2025): a way of expressing usage preferences that, as Cloudflare itself acknowledges, some companies may ignore. It is not currently part of the RFC 9309 standard.

And if you need a bot genuinely kept out, the tool is your WAF or CDN, not robots.txt: it is the only layer that can stop an agent that ignores the file or identifies itself under another name.

Step by step

How can you check which AI bots visit your site?

In five steps, with no paid tools. It will take you less than an hour the first time.

1. 1

   Open your live robots.txt

   Visit yourdomain.co.uk/robots.txt in your browser: that is what bots see, not what you think you configured in your CMS. In Google Search Console, the robots.txt report tells you which version Google has read and whether it contains errors. If your CDN serves a managed robots.txt, it may not match your site’s own file.
2. 2

   Look for the user agents in your server logs

   Download the access log (Apache or Nginx, from your hosting panel) and count each bot’s visits with the command below. You will see who comes in, how often and to which URLs.
3. 3

   Check they are who they claim to be

   User agents can be spoofed. Check the IP against the official lists: OpenAI publishes gptbot.json, searchbot.json and chatgpt-user.json; Anthropic, claude.com/crawling/bots.json; Common Crawl, ccbot.json; and Amazon and Mistral publish their own lists.
4. 4

   Review your CDN and WAF

   In Cloudflare, look at AI Crawl Control and the security events: which bots are being blocked and with which status code. If OAI-SearchBot or PerplexityBot are getting a 403, your robots.txt makes no difference.
5. 5

   Check the outcome in the answers

   Ask ChatGPT with search, Perplexity and Copilot about your service and your brand, and note whether they cite you and with which URL. Repeat every month with the same questions to track progress.

# Which AI bots have visited your site? (Apache or Nginx log)
BOTS="GPTBot|OAI-SearchBot|ChatGPT-User|Claude|Perplexity|CCBot|Bytespider"
BOTS="$BOTS|Meta-External|Meta-WebIndexer|Amazonbot|Amzn-|Applebot|MistralAI"
grep -oiE "[A-Za-z-]*($BOTS)[A-Za-z-]*" access.log | sort | uniq -c | sort -rn

Common questions

Frequently asked questions about robots.txt for AI

Verification

Sources

All consulted on 9 October 2026. Bot names and behaviour change often: if something does not add up, the official source takes precedence.

• OpenAI: Overview of OpenAI Crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot).
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Perplexity: Perplexity Crawlers
• Google Search Central: Google’s common crawlers (Googlebot, Google-Extended).
• Google Search Central: AI features and your website
• Google Search Central: Google’s user-triggered fetchers (Google-Agent).
• Apple: About Applebot
• Bing Webmaster Blog: Announcing new options for webmasters to control usage of their content in Bing Chat (22 September 2023)
• Bing Webmaster Guidelines and Search Engine Journal’s summary of the February 2026 changes
• Common Crawl: CCBot
• Meta for Developers: Meta Web Crawlers
• Amazon: Amazonbot, Amzn-SearchBot and Amzn-User
• Mistral AI: Robots
• IETF: RFC 9309, Robots Exclusion Protocol (September 2022)
• Cloudflare: press release of 1 July 2025 and AI Crawl Control documentation
• Cloudflare: Content Signals Policy (24 September 2025)
• Cloudflare: report on Perplexity’s undeclared crawlers (4 August 2025)
• Tom’s Guide: Perplexity’s response to Cloudflare (August 2025)
• heise: Mistral’s chatbot is now called Vibe
• Cloudflare: Declaring your AIndependence (3 July 2024)