Free tool
robots.txt for AI crawlers
Tick the crawlers you want to refuse and copy the file. The thing worth knowing before you tick anything: 9 of these 11 feed an assistant’s answers, and the other 2 do nothing but gather training data. Blocking one of those two changes nothing about whether you are recommended today. Blocking one of the 9 removes you from that surface.
Choose what to refuse
Nothing is ticked to begin with, because that is the state your site is already in. Every agent name here was checked against a live request rather than copied from a list.
Nothing is blocked, so this file changes nothing about where you can be recommended.
User-agent: * Allow: / Sitemap: https://delphimonitor.com/sitemap.xml
Three things this file cannot do
Worth saying plainly, because a robots.txt is often expected to do all three.
It is a request rather than a control. A crawler that ignores robots.txt is not stopped by it, and only a server rule stops that one. Every agent above is operated by a company that publishes its own agent name and states that it honours the file, which is why they are the ones listed.
It does not remove anything already learned. A model trained before you added a line carries what it carries, and a page already in an index stays there until that index is rebuilt. This governs what happens next.
And it cannot make an assistant recommend you. Being crawlable is the floor rather than the work: it is what makes you eligible to be read, not what makes you the answer. Which AI crawlers to allow goes through the reasoning, and our own measurement shows how much the assistants disagree once they can read you.