AI crawler reference
Not all AI crawlers do the same job
Blocking one costs you something completely different from blocking another, and almost every checker treats them as one list. That produces a wrong answer twice over: it warns about GPTBot, which cannot remove you from ChatGPT, and stays quiet about OAI-SearchBot, which can.
Answer crawlers
Blocking these removes you
These feed the answers people actually see. If one of these cannot reach you, you are not in that assistant's results, however good your pages are.
Googlebot
GoogleFeeds: AI Overviews, AI Mode and Search
Removes you from Google Search and from the AI answers built on it. There is no separate switch: the same crawler feeds both.
“AI is built into Search … which is why robots.txt directives for Googlebot is the control.” — Google
OAI-SearchBot
OpenAIFeeds: ChatGPT search results
Removes you from ChatGPT's search answers. ⭐ This is the OpenAI crawler that matters for visibility, and the one most audits never mention.
“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.” — OpenAI
PerplexityBot
PerplexityFeeds: Perplexity answers and citations
Removes you from Perplexity's cited results.
“designed to surface and link websites in search results … not used to crawl content for AI foundation models.” — Perplexity
Claude-SearchBot
AnthropicFeeds: Claude's web search
Removes you from what Claude finds when it searches the web.
“navigates the web to improve search result quality.” — Anthropic
AppleBot
AppleFeeds: Siri and Spotlight suggestions
Removes you from Apple's search surfaces. Worth knowing: AppleBot is one of the few that renders JavaScript.
User fetchers
Blocking these frustrates a reader
These only fetch when a person asks an assistant to go and look at a specific page. The load is negligible, and blocking them mostly frustrates somebody who was already trying to read you.
ChatGPT-User
OpenAIFeeds: A page a person asked ChatGPT to go and open
Stops ChatGPT opening your page when a user explicitly asks it to. ⚠️ OpenAI notes robots.txt may not apply to these, because the fetch is user-initiated rather than a crawl.
Claude-User
AnthropicFeeds: A page a person asked Claude to open
Stops Claude fetching your page on a user's direct request.
Perplexity-User
PerplexityFeeds: A page a person asked Perplexity to open
Stops Perplexity opening your page on request.
Training crawlers
Blocking these is a choice, not a fault
These collect data to train models rather than to answer questions. Blocking them is a legitimate publishing decision that thousands of sites make on purpose, and it does not remove you from any assistant's answers.
GPTBot
OpenAIFeeds: OpenAI model training data
⭐ Does NOT remove you from ChatGPT. This is the single most repeated mistake in AI SEO. Blocking GPTBot keeps your content out of future training runs and changes nothing about ChatGPT's search answers, which OAI-SearchBot feeds.
ClaudeBot
AnthropicFeeds: Anthropic model training data
Keeps you out of training. Claude's web search uses Claude-SearchBot, which is a separate agent.
Google-Extended
GoogleFeeds: Gemini app grounding and Google model training
⭐ Does NOT affect AI Overviews or AI Mode, and is not a ranking signal. Google says so directly. Blocking it is a decision about Gemini and about training, nothing more.
“does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.” — Google
CCBot
Common CrawlFeeds: The Common Crawl dataset, which many models train on
Keeps you out of a widely reused public dataset.
meta-externalagent
MetaFeeds: Meta model training data
Keeps you out of Meta's training data.
Bytespider
ByteDanceFeeds: ByteDance and Doubao model training
Keeps you out of ByteDance's training data. Widely blocked deliberately.
No major AI crawler runs JavaScript
Measured across a large sample of production server logs, OpenAI's, Anthropic's, Meta's, ByteDance's and Perplexity's crawlers all download script files without executing them. ChatGPT's crawler spends 11.5% of its requests on JavaScript files and runs none of them; Claude's spends 23.8%. The exceptions are Google's Gemini and Apple's AppleBot, both of which render through their own search infrastructure.
So a page can rank first on Google and be invisible to ChatGPT at the same moment.
And most of their effort is wasted on pages that are not there
ChatGPT's crawler spends 34.8% of its fetches on URLs that no longer exist, and Claude's 34.2%. Googlebot spends 8.2%. ChatGPT spends 14.4% of its requests following redirects, against Googlebot's 1.5%. That is why a stale sitemap and a chain of redirects cost you more here than they do in ordinary search: the budget they burn is budget that never reaches your real pages.
Questions about AI crawlers
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot collects training data. OAI-SearchBot is the crawler that feeds ChatGPT's search answers, and OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. Blocking training crawlers while allowing answer crawlers is a coherent position that many publishers hold on purpose.
Does Google-Extended affect AI Overviews?
No. Google states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal there. It governs Gemini app grounding and model training. AI Overviews and AI Mode are controlled by your robots.txt rules for Googlebot, which also control ordinary Search.
Which AI crawlers should I allow?
The answer crawlers, if you want to appear in AI answers: Googlebot, OAI-SearchBot, PerplexityBot and Claude-SearchBot. The user fetchers are worth allowing too, since they only run when a person asks an assistant to open your page. The training crawlers are a separate decision with no effect on whether you are cited.
Do AI crawlers run JavaScript?
Almost none of them. Measured across a large sample of production server logs, OpenAI's, Anthropic's, Meta's, ByteDance's and Perplexity's crawlers all download JavaScript files without executing them. Google's Gemini and Apple's AppleBot are the exceptions, because both render through their own search infrastructure. If your pages are assembled in the browser, most assistants receive an empty shell.
Is blocking AI crawlers bad for my site?
It depends entirely on which ones. Blocking an answer crawler removes you from that assistant's results. Blocking a training crawler removes you from a future model's training data and changes nothing about today's answers. A longitudinal audit of 14,000 web domains found that more than 28% of the most actively maintained sources in one major training corpus became fully restricted within a year, so this is a deliberate industry-wide position rather than a misconfiguration.
How do I check which crawlers my site blocks?
Read your robots.txt against the specific agent names, and then check your firewall separately, because bot protection sits in front of robots.txt and can refuse a crawler your robots.txt welcomes. Babel42's free audit does both: it reads your rules per crawler group and requests your pages as an identified crawler to see what your edge actually does.