Robots.txt tester

A crawler reads one group, not the whole file

This is the rule that catches nearly everybody, and almost every “why is my page still blocked” question comes back to it.

A crawler picks exactly one group out of your robots.txt: the one whose User-agent name is the longest that matches it. Having chosen, it ignores every other group in the file completely.

So a file with a User-agent: * group and a User-agent: Googlebot group does not apply both to Googlebot. It applies the Googlebot group and nothing else. Every Disallow in the catch-all is invisible to Googlebot, which is occasionally what somebody wanted and much more often a site accidentally opened up.

The same rule is why adding a group for a crawler is riskier than it looks: the moment User-agent: GPTBot appears with one line under it, GPTBot stops obeying everything it was obeying in the catch-all. That is why the suggested fix on this page repeats those rules rather than only adding the new one.

Then the longest rule wins

Within that one group, every Allow and Disallow whose pattern matches the path is a candidate, and the one with the longest pattern wins. Length is counted as characters written, so /folder/public/ beats /folder/ and the page is allowed.

When two matching rules are exactly the same length, the less restrictive one wins, which means Allow. Two characters matter in the patterns: * stands for any run of characters, and $ at the end pins the pattern to the end of the path, so /*.pdf$ matches a PDF but not report.pdf.html.

Anything no rule matches is allowed. robots.txt is a list of exceptions to “yes”, not a list of permissions, which is why an empty file and a missing file mean the same thing.

Blocking is not hiding

Disallow stops a page being fetched. It does not stop the address being listed. Google can and does show a URL it has never crawled, with no description under it, when other pages link to it.

To keep a page out of search results you need a noindex instruction on the page itself, and here is the trap: Google can only see that by fetching the page. Block the page in robots.txt and the noindex is never read, so the page stays listed indefinitely. The two tools pull against each other, and using both is the way to get neither.

For anything that genuinely must not be read, robots.txt is the wrong instrument entirely. It is a public file that politely asks, and a crawler that means you harm will read it as a list of the interesting directories. Put a password on it.

The AI crawlers, and which rules actually bind

The names have multiplied, and they are not interchangeable. Each vendor now runs separate crawlers for separate jobs, and blocking one does not block the others.

Training, search and user requests are three decisions. OpenAI splits them into GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic into ClaudeBot, Claude-SearchBot and Claude-User. Refusing to have your content used for training is a different choice from taking your site out of an assistant’s search results, and a single Disallow: / aimed at the wrong token does one when you meant the other.

Some names are not crawlers. Google-Extended and Applebot-Extended never fetch anything and will never appear in a server log. They are permission tokens: disallowing one withholds consent for a use of content that a different crawler already collected. Google states that Google-Extended does not affect how a page ranks in Search.

Some rules are requests rather than answers. The fetchers that run because a person asked about a specific page behave differently between vendors, and they say so themselves: Perplexity documents that Perplexity-User generally ignores robots.txt, while Anthropic states that Claude-User honours it. This tool says which, beside the result, because “blocked” means different things for different names.

Every token in the list here was taken from the vendor’s own documentation, with the date it was read, because a name is matched literally and one that is slightly wrong fails silently, exactly like a typo.