How to Check Which AI Crawlers Your robots.txt Actually Blocks

To check which AI crawlers your robots.txt blocks, take each crawler's user-agent token, find the group in your file that matches it, and read only that group's rules. Then confirm the result in your server logs, because robots.txt states what you want. It doesn't enforce it. Most people assume one blanket Disallow covers every bot. It often doesn't. Here's how the matching works, which bots do which job, and a short audit you can run on any site. Why one file gives different bots different answers RFC 9309 sets the rule. A crawler looks for groups whose User-agent line matches its token, case-insensitive. If it finds one, it follows that group and ignores the * group completely. It only falls back to * when nothing names it. Take this file: User-agent: * Disallow: /private/ User-agent: GPTBot Disallow: / User-agent: Googlebot Allow: / Run it per crawler: GPTBot has its own group, so it reads only Disallow: / and is blocked from everything. Googlebot has its own group, so it may fetch everything, /private/ included. The * rule never applies to it. ClaudeBot has no named group, so it falls back to * and may crawl everything except /private/. One file, three different answers. The blanket rule covered only one of the three bots. Inside the group a crawler follows, the longest matching path wins, and when an Allow and a Disallow match equally, the Allow wins. So read the rule for the exact URL you care about, not whichever line comes first. Know which bot does which job The major AI companies run separate bots for training, for search, and for fetching a page when a user asks. Blocking the wrong one either does nothing or takes you out of answers you wanted to be in. Company Training Search and answers Fetches on a user's request OpenAI GPTBot OAI-SearchBot ChatGPT-User Anthropic ClaudeBot Claude-SearchBot Claude-User Perplexity None listed PerplexityBot Perplexity-User Google Google-Extended (a token, see below) Googlebot None listed Three things trip people up: Blocking GPTBot doesn't remove you from ChatGPT search. OAI-SearchBot decides that. OpenAI's crawler docs list each bot and what it's for. Google-Extended isn't a crawler. Google's docs say it has no separate user-agent string. Googlebot does the crawling and reads the token as a control. You'll never see it in your logs, and blocking it doesn't remove you from Google Search. User-triggered fetchers may ignore robots.txt. When a person asks an assistant to read a page, the vendor treats that as the user's request, not a crawl. OpenAI says robots.txt rules may not apply to ChatGPT-User. The audit, step by step Fetch the live file from every hostname, for example https://example.com/robots.txt. Each subdomain needs its own file. Fetch the live file rather than reading the one in your repo, because a CDN can change what's served. Cloudflare's AI bot settings, for one, can add rules to it or block bots at the edge whatever the file says. Note the status code. Under RFC 9309, a 4xx response means crawlers may fetch everything. A 5xx or an unreachable file means they should assume they can fetch nothing. List every named group. Write down each User-agent line. For each bot you care about (GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot), find its group, or * if it has none, and read only that group. Test a real URL, not just the homepage. The answer for /blog/post can differ from the answer for /. Re-test after every edit. Adding a named group for one bot can quietly change what that bot may crawl, as the example above shows. If you'd rather not do this by hand, I built a free AI crawler access checker: enter one URL and it shows, for each major AI crawler, whether it's allowed and which rule decides. Confirm it in your server logs The file shows intent. Logs show behaviour. Search your access logs for the tokens above and read the pattern: A request for /robots.txt, then nothing for disallowed paths: a compliant crawler obeying your file. 403s or challenge pages: your server, firewall or CDN is blocking the bot. That's a separate layer from robots.txt. 200s on paths you disallowed: either you've misread which group applies, or the bot is ignoring the file. User-agent strings are easy to fake, so check that a request is genuine before you act on it. OpenAI publishes IP ranges for its bots, and Google documents a reverse DNS check for Googlebot. A common policy: be cited, don't be trained on Many sites want assistants to find and cite their pages but don't want those pages used for training. In robots.txt, that looks like this: User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended Disallow: / User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot Disallow: /private/ User-agent: * Disallow: /private/ Note the Disallow: /private/ repeated in the search group. Once those bots have their own group, they ignore *, so any path you want kept private has to be restated there. It's the same trap as in the first example. Keep the list current New AI user agents appear constantly. The community-maintained ai.robots.txt project lists 208 as of November 2026, and it can generate robots.txt and server configs for you. Compare your named groups against it every few months, and re-run the audit whenever you change the file.