The major AI crawler user-agents
These are the tokens to match on in robots.txt. Operators publish them, so the list is a matter of public record rather than measurement. The token in the left column is what you write in a User-agent line.
| User-agent token | Operator | What it does / feeds |
|---|---|---|
| GPTBot | OpenAI | Crawls pages to train OpenAI's models. Block this to opt out of training; it does not affect live ChatGPT browsing. |
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search results. Retrieval, not training — blocking it removes you from that surface. |
| ChatGPT-User | OpenAI | Fetches a page live when a user's ChatGPT prompt triggers browsing to that specific URL. On-demand, not a bulk crawl. |
| ClaudeBot | Anthropic | Anthropic's crawler for Claude. Governs whether Claude can fetch and use your pages. |
| PerplexityBot | Perplexity | Crawls and indexes pages so Perplexity can cite them in answers. Retrieval-driven — blocking it drops you from Perplexity citations. |
| Google-Extended | A robots.txt control token, not a separate bot. Governs whether Googlebot's existing fetches may be used to train Gemini and Vertex AI. Does not affect Search ranking. | |
| Applebot-Extended | Apple | The same pattern as Google-Extended, for Apple Intelligence. Controls training use of pages Applebot already fetches for Siri and Spotlight. |
| Bingbot | Microsoft | Bing's index feeds Copilot and ChatGPT's Bing-backed browsing, so this classic crawler is now an AI-answer input as well. |
| CCBot | Common Crawl | Builds the open Common Crawl web archive, which many model builders use as a training corpus. Blocking it opts you out of that shared dataset. |
Tokens and operators are published by the companies themselves. If a token you see in your logs is not on this list, look it up rather than guess — new ones appear and old ones get renamed.
Two jobs: training vs. retrieval
The list splits cleanly by purpose. Training crawlers (GPTBot, CCBot, and the -Extended tokens) fetch pages so a model can learn from them offline. Retrieval crawlers (OAI-SearchBot, PerplexityBot, ChatGPT-User, and Bing's index behind Copilot) fetch pages so an engine can quote or cite them while answering a live question.
That distinction is the whole game for policy. Blocking a training crawler is a data-licensing decision; it says nothing about whether the model can cite you today. Blocking a retrieval crawler is a visibility decision — it directly removes you from that engine's answers. People conflate the two constantly and then wonder why blocking GPTBot did not make them disappear from ChatGPT search, or why blocking PerplexityBot did.
robots.txt policy: allow by default
Being retrievable is the product. If an AI engine can fetch your page, it can cite you; if it cannot, you are invisible on that surface no matter how good the page is. So the sane default is to allow the retrieval crawlers and block only the specific tokens you have a deliberate reason to exclude.
The most common deliberate block is training-only: allow OAI-SearchBot and PerplexityBot so you stay citable, but disallow GPTBot and CCBot if you do not want your pages in training corpora. That keeps you in live answers while opting out of model training. A blanket 'Disallow: /' for every AI token is almost always a mistake made by copying a hostile robots.txt template — it removes you from the exact answers you are trying to win.
Match tokens exactly and case-consistently in User-agent lines. A rule under the wrong token silently does nothing.
Common questions
Does blocking GPTBot remove me from ChatGPT answers?+
No. GPTBot is OpenAI's training crawler. Live ChatGPT browsing and search use ChatGPT-User and OAI-SearchBot. Blocking GPTBot only opts you out of training — you stay citable in ChatGPT's live answers unless you also block those retrieval tokens.
Is Google-Extended a separate crawler?+
No. It is a robots.txt control token, not a bot with its own fetches. Googlebot still crawls your site for Search; Google-Extended only governs whether those fetches may be used to train Gemini and Vertex AI. Blocking it does not affect your Search ranking.
Should I block AI crawlers by default?+
Generally no. Being retrievable is what makes you citable, so the default is to allow the retrieval crawlers. Block only the specific tokens — usually training-only ones like GPTBot or CCBot — where you have a deliberate reason to opt out.
keep reading
See what AI says about your brand right now — free, on a live engine.
Scan your domain