Two kinds of crawler, two very different costs
The single most common mistake is treating "AI crawlers" as one thing. They aren't. A training crawler reads your pages to build a dataset for a future model — the effect of blocking it is deferred and diffuse. A retrieval crawler reads your page at the instant a user asks a question, so blocking it removes you from the live answer immediately and visibly.
If you only remember one distinction, make it this one. Blocking training crawlers is a slow, philosophical lever. Blocking retrieval crawlers is a fast, concrete one — and it's the one people trip over when they paste a blanket block rule into robots.txt without reading the user-agents.
| Crawler type | Cost of blocking | When blocking is right |
|---|---|---|
| Training (GPTBot, Google-Extended, CCBot) | You're excluded from future model training. Little to no effect on whether you're cited today. | You object to your work training models, or the content is proprietary and you don't want it reproduced. |
| Retrieval / browsing (OAI-SearchBot, ChatGPT-User, PerplexityBot, Bingbot) | You go invisible in live AI answers now. The assistant can't fetch you, so it can't cite you. | Content is paywalled, gated, or genuinely not meant for public retrieval. Rarely right if you want AI recommendations. |
| Abusive / high-volume (any bot ignoring rate limits) | You lose that bot's traffic — usually acceptable if it's degrading your site. | The crawler is hammering your bandwidth or ignoring robots.txt. Block or rate-limit regardless of type. |
User-agent strings and behavior change over time — verify the current ones before writing a rule. See what-are-ai-crawler-user-agents.
When blocking is legitimately the right move
There are honest reasons to block, and pretending otherwise is a slogan, not advice. If your content is paywalled or proprietary — pricing you negotiate per client, subscriber-only research, a members' archive — you may not want it reproduced in an answer you don't control. That's a real rights position, and blocking retrieval crawlers is a coherent way to hold it.
Bandwidth abuse is the least ambiguous case. A bot that ignores your crawl-delay and floods your origin is a cost problem, and rate-limiting or blocking it is just operations, independent of whether it's an AI crawler at all.
And some publishers block training crawlers as a deliberate stance — they'd rather not feed models on principle. That's legitimate. Just scope it to the training user-agents so you don't quietly delete yourself from live answers in the process.
A decision framework
Start from the goal. If you want AI assistants to find, quote, and recommend you, allow the retrieval crawlers — full stop. That's the product. Then decide training separately: allow it if you don't mind feeding models, block it if you do; either way it barely touches today's citations.
If you want to be private or protect specific content, block at the path level rather than site-wide where you can — keep public marketing pages retrievable while gating the proprietary sections. And if a bot is simply abusing your site, throttle it regardless of category. The failure mode to avoid is a blanket block written to make a training-data point that silently takes you out of the live answers you actually wanted to win.
Whether a page is retrievable is measurable on the frozen five-engine panel (ChatGPT, Claude, Gemini, Perplexity, Google AI Overviews). If you block retrieval crawlers, the drop shows up there.
Common questions
Will blocking GPTBot stop ChatGPT from citing my site?+
Mostly no. GPTBot is a training crawler — blocking it keeps your content out of future model training but has little effect on whether you're cited in a live answer. Live ChatGPT answers are fetched by OAI-SearchBot and ChatGPT-User; block those and you disappear from answers now.
If I want AI to recommend my business, should I block any AI crawlers?+
Generally not the retrieval crawlers — being retrievable is what lets an assistant find and cite you. You can still block training crawlers as a separate rights decision without hurting today's citations. Just don't block the retrieval bots by accident.
When is blocking AI crawlers actually the right call?+
Three clear cases: paywalled or proprietary content you don't want reproduced, a bot abusing your bandwidth, or a deliberate stance against your work training models. For the last one, scope the block to training user-agents so you stay visible in live answers.
keep reading
See what AI says about your brand right now — free, on a live engine.
Scan your domain