Which AI crawlers to allow, and what each one decides

GPTBot and OAI-SearchBot both belong to ChatGPT and they do opposite things. One collects text that may train a future model. The other builds the index the assistant searches while it is answering a buyer. Refusing the first changes nothing about the answer somebody gets this afternoon. Refusing the second is the closest thing there is to opting out of that assistant.

The third kind is not a crawl at all. It is one fetch, made because somebody has just asked a question your page might answer, and several operators document that the rules in your file may not apply to it.

So there is no single answer to whether to allow AI crawlers. There are three answers, and only one of the three is a decision a commercial team would want to take twice.

• GPTBot. ChatGPT. Documented as collecting content to make the foundation models more useful and safer.
• ClaudeBot. Claude. Documented as collecting web content that could contribute to training.
• meta-externalagent. Documented as crawling for uses such as training foundation models and improving products.
• Amazonbot. Documented as improving Amazon products and services, and its content may be used to train models.
• CCBot. Common Crawl. A public archive rather than one company's model, and an input to many of them.

Refusing all five is a defensible position and it is not a visibility decision. A model that has already read you does not forget on the day you add the line, and nothing in this group changes the answer a buyer gets this month. What it changes is what the next model is built from.

• OAI-SearchBot. Documented as surfacing websites in ChatGPT search features.
• Claude-SearchBot. Documented as improving the relevance and accuracy of search responses.
• PerplexityBot. Documented as surfacing and linking websites in Perplexity results, and not used for training.
• Googlebot. The fetcher behind Google Search, and the content Google-Extended separately governs for Gemini training and grounding.
• Applebot. The fetcher behind Apple search features. Apple states that pages disallowing Applebot-Extended can still appear in results.
• Amzn-SearchBot and meta-webindexer. Search inside Amazon products, and Meta AI search results.

These are the ones to allow unless there is a specific reason not to. They decide whether an assistant can read your own words about your own product, or can only read what other sites say about you.

• ChatGPT-User. OpenAI documents that because the action is initiated by a person, robots.txt rules may not apply.
• Perplexity-User. Documented as generally ignoring robots.txt, on the same reasoning.
• meta-externalfetcher. Documented as able to bypass robots.txt because the request is made at a user's request.
• Amzn-User. Documented as possibly not following all robots.txt directives, for the same reason.
• Claude-User. Anthropic documents the opposite: disallowing it stops Claude retrieving your content in response to a user query.

That disagreement is a fact about the standard rather than a fault in your configuration. A live fetch is not something you turn off with a line, and it is not something you reliably turn on with one either.

The consequence is usually met the wrong way round. A refused crawler does not make you invisible to that assistant. It makes you describable only by other people. The assistant still answers, still names brands, and still says something about you, drawn from what it was trained on and from whatever else on the web mentions you. What it cannot do is read the page where you corrected the thing it is getting wrong.

So the ceiling is not on your visibility. It is on your ability to move it. Every step after the first is bounded by a setting made above it, which is why a refusal that suppresses several steps at once is sequenced ahead of the weakest step, however weak that step looks.

It is also the cheapest thing on any plan to fix. Every other move takes a page, a publisher or a quarter. This one takes a line, and until it is fixed the work that follows it is being done against a cap.

The file is a statement of intent. The edge is the enforcement, and it is a different system, usually owned by a different team. A rule written to keep scrapers off a pricing page does not know it is also refusing the crawler that builds an assistant's index.

It is hard to see because it does not look like a refusal. A challenge page answers with a success code and a body, so anything checking only the status code reads it as your home page, and then reads your home page as carrying nothing. Delphi counts a challenge page as a refusal for that reason: scoring one as a home page would invent a finding about a site that is fine.

And the rule may not be about crawlers at all. One refusal Delphi investigated turned out to rest on the shape of the connection rather than on who was making it: the same address in the same minute, refused through one client and served through another. A rule that refuses AI crawlers and a rule that refuses anything not shaped like a browser produce the same symptom and need different fixes.

• Ask as the crawler. Fetch a real page with each token as the user agent. A 403 is a refusal. So is a success code carrying a challenge page, which is the one most checks miss.
• Read the file the way a crawler reads it. A group naming a token wins outright over the wildcard group, so a permissive rule for everybody does not rescue a token that is named and disallowed elsewhere in the same file. An empty Disallow allows everything, while a Disallow of a single slash blocks everything.
• Look for arrivals. A token turning up in your own logs is the only evidence that everything in front of the file agreed with the file.

A brand that has only ever run the second of those has checked the half of this that was never the problem.