How do I block GPTBot in robots.txt?

A `User-agent: GPTBot` group with `Disallow: /` under it. That closes training access only. It does not affect ChatGPT's search surface, which is served by OAI-SearchBot under a separate user-agent. Blocking both takes you out of today's answers and out of what future models will know — two decisions worth making separately rather than together.

If I block Google-Extended, do I drop out of Google Search?

No. Google's documentation states explicitly that Google-Extended does not affect inclusion in Google Search and is not a ranking signal. Apple's setup has the same shape: Applebot-Extended does not crawl at all — it governs whether data already collected by Applebot may be used to train generative models — and pages that disallow it can still appear in search results.

Is noindex enough, or do I also need a robots.txt rule?

They do different jobs and do not substitute for each other. `noindex` addresses a search index: a request not to show the page in results. It means nothing to a training collector, which is not indexing anything — it is reading text. To keep a page out of model training you need the robots.txt rule; to keep it out of a results page you need noindex. Neither works in the other direction.

Why isn't Disallow enough on a page that is already noindex?

Because together they can defeat each other. If you disallow a page, the crawler does not fetch it — so it never READS the noindex tag. Pages in that state typically get stuck as "indexed, though blocked". If the goal is removal from search, allow the crawl and let the noindex be seen. If the goal is exclusion from training, Disallow is the right tool — but then accept that nothing else on the page gets read either, including its canonical.

How do I know a bot is really who it claims to be?

You don't, from the name alone — a user-agent string is free text. If you intend to rely on a bot's identity, the published IP ranges and forward-confirmed reverse DNS are the only checks that prove anything; Common Crawl, for instance, warns explicitly that crawlers falsely identifying themselves as CCBot exist. None of this is needed for robots.txt itself, which addresses names, and anyone honouring it is using their own.

Cookbook

AI crawlers in robots.txt: which to allow, which to block

There are three classes of AI bot in your logs, not two — and most block lists never reach the third.

Updated · August 9, 2026

Most robots.txt advice for AI treats the question as binary: allow the bots or block them. That framing loses the decision. What actually shows up in your logs falls into at least three commercially distinct classes: training crawlers, which feed what future models will remember about you; retrieval crawlers, which look for sources to cite in answers being generated today; and user-initiated fetchers, which arrive because a specific person asked a specific question. Block all three with one gesture and you also opt out of the one that could have sent you a reader this afternoon. This recipe walks the decision end to end: what is actually on the list, what each of the three scenarios costs you, how to check that your rule matches anything at all — and two dated mistakes of our own, which are the most useful part.

The user-agent list

Every bot below is named in its own vendor's documentation, with the source URLs numbered at the end of the page. Anything we could not confirm against an official source was left out — a user-agent list assembled from blog posts gives exactly the false confidence this recipe argues against.

The user-agent list
User-agent	Vendor	Class	What matters about it
GPTBot	OpenAI	training	Collects content for foundation-model training. The one most block lists start with.
OAI-SearchBot	OpenAI	retrieval	Powers ChatGPT's search features. Blocking it removes you from answers, not from training.
ChatGPT-User	OpenAI	user-initiated	Fetches when a person asks. Documented as potentially outside robots.txt scope.
ClaudeBot	Anthropic	training	Collects web content that could contribute to model training.
Claude-SearchBot	Anthropic	retrieval	Navigates the web to improve search result quality.
Claude-User	Anthropic	user-initiated	Visits pages in response to a user's question.
PerplexityBot	Perplexity	retrieval	Indexes and links. Documented as NOT used to crawl for AI foundation models.
Perplexity-User	Perplexity	user-initiated	The docs state plainly: this fetcher generally ignores robots.txt rules.
Google-Extended	Google	training	Controls use only. Blocking it does NOT affect Google Search inclusion or ranking.
Applebot-Extended	Apple	training	Does not crawl. It governs how Applebot's data may be used. You can still appear in search.
CCBot	Common Crawl	open corpus	Open archive that feeds many smaller models. A separate opt-out registry also exists.
Meta-ExternalAgent	Meta	training	Crawls for use cases such as training foundation AI models.

Last verified · August 9, 2026

The procedure

Decide per class, not in one gesture
Write down three separate answers: do you want to be part of what future models know (training), do you want to be citable in answers generated today (retrieval), and may an assistant open one of your pages when a user explicitly asks (user-initiated). These answers are rarely the same. For a young brand, training access is the long-horizon investment and retrieval is this quarter's traffic; blocking both is two different losses on two different clocks, and only one of them is recoverable later.
Write path-explicit rules
In robots.txt, `*` spans the slash. This is the quiet failure mode: `Disallow: /*/demo` matches `/en/demo` and also `/en/products/demo`. That is not hypothetical for us. A rule of exactly that shape walled four live product pages off from every AI crawler on our own site — pages that exist specifically to be ingested by AI. Spell the locale and the full path out, or anchor the pattern with `$`. If one rule needs to cover several things, write several lines; brevity buys you nothing here.
Bind the rule to the route, not to the path shape
robots.txt is a static file, so every path in it is a COPY of something that lives in your code. Rename the page and the copy does not follow: the rule stays, but it now matches nothing — and a rule that matches nothing raises nothing. The block still looks present in review, the diff is clean, the tests stay green, and the renamed page goes out to every collector on the internet. We measured this on ourselves too. The fix is a check that reconciles the paths in robots.txt against the routes that actually exist, and fails when a blocked path has no page behind it.
Verify that the rule matches anything
Do not eyeball it. Take a robots.txt parser — most languages have one, and Google published the logic of its own — and ask two questions of every URL you care about: can Googlebot reach it, and can the bot you meant to block reach it. Ask them separately, because the common mistake is not that you failed to block something; it is that you blocked MORE than you thought. The longest matching rule wins and `Allow` beats `Disallow` at equal length, which is hard to compute in your head and trivial to compute in code.
Check the server logs for who actually came
robots.txt is a statement of intent; the log is evidence. Filter on the user-agent field and look at which bots request what, how often, and what status code they get back. Two things surface that you cannot learn anywhere else: whether your rule holds in practice, and whether anything is arriving under a borrowed name. User-agent strings are free text, so if you intend to act on a bot's identity — rate-limiting it, or logging it as a signal — published IP ranges or forward-confirmed reverse DNS are the only checks that prove anything.
Know what robots.txt cannot do
It does not reach most user-initiated fetches. Perplexity's documentation says outright that its user fetcher generally ignores robots.txt, and OpenAI notes that ChatGPT-User may operate outside its scope. That is a deliberate distinction rather than a violation: a person asking to open a page is not a crawl. If content genuinely must not leave your site, its place is behind authentication, not behind a robots.txt line. robots.txt governs automated collection, and only by those who choose to honour it.

Frequently asked questions

How do I block GPTBot in robots.txt?: A `User-agent: GPTBot` group with `Disallow: /` under it. That closes training access only. It does not affect ChatGPT's search surface, which is served by OAI-SearchBot under a separate user-agent. Blocking both takes you out of today's answers and out of what future models will know — two decisions worth making separately rather than together.
If I block Google-Extended, do I drop out of Google Search?: No. Google's documentation states explicitly that Google-Extended does not affect inclusion in Google Search and is not a ranking signal. Apple's setup has the same shape: Applebot-Extended does not crawl at all — it governs whether data already collected by Applebot may be used to train generative models — and pages that disallow it can still appear in search results.
Is noindex enough, or do I also need a robots.txt rule?: They do different jobs and do not substitute for each other. `noindex` addresses a search index: a request not to show the page in results. It means nothing to a training collector, which is not indexing anything — it is reading text. To keep a page out of model training you need the robots.txt rule; to keep it out of a results page you need noindex. Neither works in the other direction.
Why isn't Disallow enough on a page that is already noindex?: Because together they can defeat each other. If you disallow a page, the crawler does not fetch it — so it never READS the noindex tag. Pages in that state typically get stuck as "indexed, though blocked". If the goal is removal from search, allow the crawl and let the noindex be seen. If the goal is exclusion from training, Disallow is the right tool — but then accept that nothing else on the page gets read either, including its canonical.
How do I know a bot is really who it claims to be?: You don't, from the name alone — a user-agent string is free text. If you intend to rely on a bot's identity, the published IP ranges and forward-confirmed reverse DNS are the only checks that prove anything; Common Crawl, for instance, warns explicitly that crawlers falsely identifying themselves as CCBot exist. None of this is needed for robots.txt itself, which addresses names, and anyone honouring it is using their own.

Sources

OpenAI — Bots (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot). https://developers.openai.com/api/docs/bots
Anthropic — Does Anthropic crawl data from the web? https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
Perplexity — Perplexity Crawlers. https://docs.perplexity.ai/docs/resources/perplexity-crawlers
Google — Google crawlers (Google-Extended). https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
Apple — About Applebot. https://support.apple.com/en-us/119829
Common Crawl — CCBot. https://commoncrawl.org/ccbot
Meta — Web crawlers. https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/

AI crawlers in robots.txt: which to allow, which to block

The user-agent list

The procedure

Decide per class, not in one gesture

Write path-explicit rules

Bind the rule to the route, not to the path shape

Verify that the rule matches anything

Check the server logs for who actually came

Know what robots.txt cannot do

Frequently asked questions

Sources