A robots.txt that allows GPTBot does not mean GPTBot can reach your site

I built SeenLens, a one-time AI readiness audit for websites. I wanted to know what happens when a page's published crawler rules and its HTTP responses tell different stories. I scanned one public URL on each of 30 sites. In that run, 5 returned HTTP 403 or 202 to the scanner. I classified 4 as bot-wall responses based on their headers and body size: a challenge header on one response, bot protection vendor headers on two HTTP 403 responses, and an empty body on one HTTP 403 response. The sites are anonymous. These were requests from our scanner's network, so the results describe what the scanner received at the time of the test. Two checks, two different answers robots.txt is a policy file. A crawler that follows the Robots Exclusion Protocol uses it to decide which paths it should request. It checks the rule group for its crawler token and the path of the URL. A crawler that finds no matching rule in its applicable group can fetch that path under the protocol. Read the matching rules in RFC 9309. The server makes a separate decision when a request arrives. A content delivery network (CDN) or web application firewall (WAF) can return an error, rate-limit the request, or send a challenge page. A challenge asks the client to do something, such as run JavaScript, before it receives the real page. The two checks can disagree. A server can send the page even if robots.txt disallows a crawler. A server can also block a request even if robots.txt allows it. Check both the policy and the response for the exact URL you care about. The SeenLens robots.txt guide walks through rule groups and paths. Test one URL from your terminal Use a site you control. Start with robots.txt on the same scheme and host as the page. The first command lists likely matches so you know where to look. Open the full file before deciding which group applies. This short command can miss wildcard rules and longer groups. curl -sS 'https://example.com/robots.txt' | grep -iE -A 3 'user-agent:.*(gptbot|oai-searchbot|claudebot|\*)' Use GET for the next two requests. Some servers handle HEAD differently. Save the headers and body so you can tell the real page from a challenge. Replace the example URL with your own page. curl -sS -D /tmp/crawler.headers -o /tmp/crawler.html -w '%{http_code}\n' \ -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)' \ 'https://example.com/' curl -sS -D /tmp/browser.headers -o /tmp/browser.html -w '%{http_code}\n' \ -A 'Mozilla/5.0' 'https://example.com/' Each command prints the status code that the site returned. Compare the two codes, then look for the page title or a phrase from the main content in each body. If you work behind an HTTPS proxy, the saved header file starts with the proxy's own status line, so read the status line that follows it. A redirect may explain the status. If so, examine the redirect target and repeat the two requests there. A different response size is a clue, especially when one response is a short error page. Size can also vary because of personalization, so read the bodies before you decide. Here is an illustrative comparison. The output is an example of what a challenge might look like. browser.headers: HTTP/2 200 crawler.headers: HTTP/2 403 crawler.headers: cf-mitigated: challenge Cloudflare documents that header for its challenge responses. Other providers use different signals. A 403 by itself does not identify the rule or provider. Cloudflare challenge response documentation Changing the user agent changes the label you send. It does not change your IP address. A verified crawler can receive a different response from the same server. Use firewall logs and the vendor's published verification method when the identity of a real request matters. What the 30-site run showed The run used SeenLens rules 0.4.1 on 7 November 2026, between 09:06 and 09:20 UTC. The table groups the scanner's page responses by HTTP status. All 30 scans finished and were scored. Scanner response Pages Score range HTTP 200 25 65–96 Other response or fetch failure 5 (4 × HTTP 403, 1 × HTTP 202, no fetch failures) 17–49, none unscored The median score was 79 for the HTTP 200 group and 40 for the other five. Five pages did not return HTTP 200. One returned HTTP 202 with an x-amzn-waf-action: challenge header. It is the only response I count as a confirmed challenge, because it is the only one with an explicit challenge signal. Two returned HTTP 403 with bot protection headers (server: DataDome and x-datadome: protected) and a body under 1 KB. Those headers identify the vendor. The scan did not keep the body, so I could not confirm a challenge on those two. One returned HTTP 403 with an empty body and no vendor header. The last one returned HTTP 403 with a short page that said the request was blocked and asked the visitor to log in. It had no vendor header, and the scanner did not classify it as a bot wall. On one of the two pages with bot protection headers, robots.txt allowed all 13 AI crawler tokens tested at the scanned path, but the page request still got HTTP 403. The scanner caps the score of a page it cannot read at 49, so these five pages scored between 17 and 49. A blocked fetch makes some page checks impossible. Keep those checks separate from failures on pages that the scanner could read. A blocked fetch also cannot establish what a real AI crawler received from the same site. These results are a snapshot of the URLs, client and time in this run. Decide which crawlers you want to allow Start with the job each crawler does. OpenAI documents OAI-SearchBot for search and GPTBot for content that may be used to train models. Those crawler controls are independent. Anthropic also documents separate search and training crawlers. If you intend to allow search indexing while opting out of training, review each operator's current crawler tokens and policy. OpenAI crawler documentation · SeenLens guide to OpenAI crawlers Put the policy you intend in robots.txt. Then check whether your CDN or firewall makes the same access decision. Start with the security events for the test URL and time. Look for a bot rule, a rate limit or a managed challenge. If you change a rule, narrow its scope to the intended crawler and page. Use verified crawler identities or published IP ranges when your provider supports them. Anyone can send a crawler name in a request header. SeenLens firewall troubleshooting guide After a change, fetch the URL again and inspect what the client gets. For sites where you operate the firewall, check its logs for the next verified crawler request. If you want a quick first pass, the SeenLens AI crawler checker reads rules for 14 crawler tokens and compares a browser response with four crawler user agents. It is free and needs no signup.