Should You Block or Allow AI Crawlers? GPTBot, ClaudeBot and the Trade-Offs

Your legal team wants AI companies blocked from your content. Your marketing team wants your brand cited in ChatGPT answers. Both requests land on the same file: robots.txt. This post explains what the directives actually do, what you gain and lose either way, and how to make the decision defensibly rather than reactively.

Which AI crawlers are we actually talking about?

Not all AI-related bots do the same job. Treating them as one category is the most common mistake we see in crawler policy. Three distinct functions matter:

• Training crawlers collect content used to train future models. Your content is absorbed into model weights and cannot be attributed back to you at answer time.
• Retrieval and search crawlers build indexes that assistants query live when answering a user’s question. This is the path that produces a visible citation with your URL.
• User-triggered fetchers retrieve a specific page because a user pasted a link or the assistant decided to read a named source in real time.

The main user-agents you’ll encounter:

• GPTBot — OpenAI’s crawler used for model training data collection.
• OAI-SearchBot — OpenAI’s crawler for its search index, which is what surfaces cited links in ChatGPT.
• ChatGPT-User — fetches a page live when a user’s request requires it.
• ClaudeBot — Anthropic’s crawler for collecting web content.
• Claude-User and Claude-SearchBot — Anthropic’s user-triggered fetch and search-index agents.
• PerplexityBot — indexes pages so they can be surfaced and linked in Perplexity answers.
• Google-Extended — not a crawler at all. It’s a control token that tells Google whether content already crawled by Googlebot may be used for Gemini and related generative training. Blocking it does not affect Google Search ranking.
• CCBot — Common Crawl, a non-profit public web archive that many organisations, including AI labs, use as a training corpus input.
• Applebot-Extended — the equivalent generative-use opt-out for Apple’s crawler.

The critical point: blocking the training crawler and blocking the search crawler produce completely different business outcomes.

How robots.txt directives actually work

Robots.txt is a voluntary protocol served at the root of your domain. It is a request, not an enforcement mechanism. Compliant crawlers read it and obey; non-compliant scrapers ignore it entirely. If your objective is genuine technical prevention, robots.txt is the wrong tool — you need server-level or WAF-level blocking by user-agent and IP range.

Directives are matched by user-agent group. A crawler obeys the most specific group that names it, and only that group. This trips people up constantly: if you’ve a group for GPTBot and a separate User-agent: * group, GPTBot reads only its own group and ignores the wildcard rules entirely.

A block of training crawlers while remaining eligible for citation looks like this:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

A selective block, protecting only proprietary material while leaving public content open:

User-agent: ClaudeBot
Disallow: /research-library/
Disallow: /client-portal/
Allow: /

Two practical cautions. First, robots.txt is public — anyone can read it and infer which directories you consider valuable. Second, blocking a crawler does not remove content it has already collected.

What do you actually gain by blocking?

The honest case for blocking rests on three arguments, and they aren’t equally strong across organisations.

• Content that’s the product. If your paid research, courseware, or data is your revenue line, having it absorbed into a model that answers the same questions for free is a direct commercial threat.
• Contractual and confidentiality obligations. Some client agreements or regulatory positions restrict how material may be reproduced or redistributed. Blocking is a documentable control.
• Negotiating position. Publishers with substantial archives have used access as leverage in licensing conversations. This only works at scale.

Server load is occasionally cited but is rarely material for a normal corporate site.

What do you lose?

The cost is specific and it compounds. If you block the retrieval and search crawlers, your pages become ineligible for citation in the assistants that use them. When a buyer asks an assistant to compare vendors in your category, the answer is assembled from sources the model can retrieve. If yours is not retrievable, a competitor’s is — and the assistant will still answer the question. You don’t remove yourself from the conversation; you remove yourself from the citation.

There’s a second-order cost. Assistants increasingly synthesise what a company does from whatever sources are available. If your own site is blocked, the description of your business gets built from directories, aggregators, review sites and press coverage you don’t control. Blocking doesn’t create silence. It transfers narrative control to third parties.

A decision framework you can defend

Run four questions in order. The answers, not the ideology, should determine the policy.

1. Is the content itself the revenue-generating asset, or is it demand generation for something else? Gated research libraries, paid datasets and licensed archives argue for blocking. Marketing content, service pages, methodology explainers and thought leadership exist to be found — blocking them defeats their purpose.
2. Separate training from retrieval. For most organisations the defensible middle position is to opt out of training-only crawlers while explicitly allowing search and retrieval agents. You reduce uncompensated absorption of your archive while staying eligible to be cited.
3. Segment by directory, not by domain. Very few organisations need a site-wide decision. Block the proprietary sub-directories, open the public ones.
4. Is robots.txt sufficient for your risk level? If the real requirement is preventing access rather than expressing a preference, put the content behind authentication. A compliance obligation isn’t met by a text file that bad actors ignore.

Review it like a policy, not a config file

Crawler names change, new agents appear, and vendors split single bots into separate training and search agents. A robots.txt written eighteen months ago is probably blocking a bot that no longer exists and silently allowing three that do. Put it on a quarterly review, log which agents are hitting your server, and check that what you intended to block is actually what is blocked.

Whatever you decide, measure the effect. Our ARIA citation tracker shows whether your pages are being cited in AI answers, which is the only reliable way to see whether a crawler policy change moved the needle. If you are building the policy from scratch, our AI visibility practice covers how the crawler decision fits the wider strategy.

Frequently asked questions

Does blocking GPTBot remove my content from ChatGPT?

No. Blocking GPTBot stops future collection for training. Content already collected remains in existing models, and ChatGPT can still reach your pages through its separate search and user-fetch agents unless you block those too.

Will blocking Google-Extended hurt my Google Search rankings?

No. Google-Extended governs whether crawled content may be used for generative AI products. It’s separate from Googlebot and has no effect on classic search indexing or ranking.

Can I block AI crawlers but still appear in AI answers?

Partly. If you block only training crawlers and allow search and retrieval agents, you remain eligible for citation. If you block the retrieval agents, you become ineligible for citation in those systems.

Is robots.txt legally binding?

It is a voluntary standard, not a contract or a technical control. Well-behaved crawlers honour it. If you need enforceable prevention, use authentication, server-level blocking, or terms of use combined with both.