Free tool
robots.txt for AI crawlers
Tick the crawlers you want to refuse and copy the file. The thing worth knowing before you tick anything: of these 18, 7 decide whether an assistant can read your own pages when it answers, 6 only gather or govern training data, and 5 fetch a page because a person has just asked about it. Refusing a training crawler changes nothing about whether you are recommended today. Refusing one of the 7 stops that surface reading your own words about your own product.
Choose what to refuse
Nothing is ticked to begin with, because that is the state your site is already in. Every description comes from the operator's own documentation, linked on its row, and every agent name was checked against a live request.
- OpenAI’s documentation
- OpenAI’s documentation
- OpenAI’s documentation
- Anthropic’s documentation
- Anthropic’s documentation
- Anthropic’s documentation
- Perplexity’s documentation
- Perplexity’s documentation
- Google’s documentation
- Microsoft’s documentation
- Meta’s documentation
- Meta’s documentation
- Meta’s documentation
- Amazon’s documentation
- Amazon’s documentation
- Amazon’s documentation
- Common Crawl’s documentation
- Apple’s documentation
Nothing is blocked, so this file changes nothing about where you can be recommended.
User-agent: * Allow: / Sitemap: https://delphimonitor.com/sitemap.xml
Three things this file cannot do
Worth saying plainly, because a robots.txt is often expected to do all three.
It is a request rather than a control. A crawler that ignores robots.txt is not stopped by it, and only a server rule stops that one. Every agent above is published by its own operator so that a site can name it in this file. The operators of ChatGPT-User, Perplexity-User, meta-externalfetcher and Amzn-User also document that those agents, which fetch on a person’s behalf, may not follow it, so a line refusing one of them is a request its operator has said it may not honour.
It does not remove anything already learned. A model trained before you added a line carries what it carries, and a page already in an index stays there until that index is rebuilt. This governs what happens next.
And it cannot make an assistant recommend you. Being crawlable is the floor rather than the work: it is what makes you eligible to be read, not what makes you the answer. The work above that floor is usually called generative engine optimisation. Which AI crawlers to allow goes through the reasoning, and our own measurement shows how much the assistants disagree once they can read you.