01What is this tester for?
The robots.txt file tells bots what they may crawl. One badly written rule and the whole site vanishes from Google — or, increasingly, from the answers of ChatGPT, Claude and Perplexity: many sites block AI crawlers without knowing it, through a generic rule or a copy-pasted firewall setting. This tool reads the file live and returns the verdict bot by bot, with the exact rule that applies.
The subtlety lies in the priorities: each bot picks the most specific User-agent group that concerns it, then the longest rule wins, with Allow prevailing on ties. A Disallow: / under User-agent: * therefore does NOT block Googlebot if a User-agent: Googlebot group exists above. Pair this with our page audit, which also checks robots.txt and sitemap.
02Understanding the directives
- User-agent
- Opens a rule group for one bot (User-agent: Googlebot) or all (User-agent: *). A bot obeys a single group: the most specific one for it.
- Disallow
- Forbids paths starting with this prefix. Disallow: / blocks everything; an empty value forbids nothing.
- Allow
- Re-allows a path inside a forbidden area (Allow: /blog under Disallow: /). At equal length, Allow beats Disallow.
- * and $
- * matches any sequence of characters, $ anchors the end (Disallow: /*.pdf$ blocks PDFs only).
- Sitemap
- Declares the XML sitemap's URL — the only directive read outside any group, by all bots.
- Crawl-delay
- Request spacing asked of some bots. Ignored by Google; Bing honours it.
03Block or allow AI crawlers?
Blocking GPTBot, ClaudeBot or Google-Extended protects your content from model training — but also removes your pages from AI assistants' answers, a growing share of site discovery. Note the difference between training bots (GPTBot, ClaudeBot) and search or user-action bots (OAI-SearchBot, ChatGPT-User, Claude-User): you can block the former and allow the latter.
Whatever your choice, make it consciously: check here what your file actually says, and keep in mind that robots.txt is a convention, not a barrier — a misbehaved bot can ignore it. For actual blocking, act at the server or firewall level (WAF rules, AS blocking).
04Frequently asked questions
No robots.txt: is that a problem?
No. Without a robots.txt (404 response), all bots consider everything allowed — a perfectly valid choice for a fully public site. The file becomes useful when you want to exclude areas (admin, internal search, carts), declare your sitemap, or control AI bots. However, a robots.txt answering with a 5xx error is a real problem: Google pauses crawling the site.
Does robots.txt keep a page out of Google?
Not exactly: it prevents crawling, not indexing. A page blocked by robots.txt can still appear in results (without a description) if other sites link to it. To exclude a page from the index you need a meta robots noindex tag — and thus, paradoxically, to allow the bot to visit the page so it can see it.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
Three distinct uses at OpenAI: GPTBot collects for training future models; OAI-SearchBot indexes for ChatGPT's built-in search; ChatGPT-User visits a page at a user's request (when they paste your URL into the conversation). You can block training while staying visible in search: Disallow for GPTBot, Allow for the other two. Anthropic offers the same granularity with ClaudeBot and Claude-User.
My robots.txt blocks a page, but it is already in Google. What now?
The robots.txt block removes nothing from the index — it even prevents Google from coming back to see the noindex tag you may have added. The correct sequence: unblock the path in robots.txt, add noindex to the page, wait for de-indexing, then re-block if needed. In an emergency, Search Console's removal tool takes the URL down within hours (temporarily).
Are the rules case-sensitive?
Paths, yes: Disallow: /Admin does not block /admin. Bot names, no: user-agent: googlebot and User-agent: Googlebot are equivalent. Also mind URL encoding: the rules apply to the path as it appears in requests, parameters included if you mention them.
05More tools
Web & SEO
Page auditAudit a web page: security, SEO, performance — score /100 HTTP headersSee the HTTP headers your browser sends Security headersGrade a site's security headers: CSP, HSTS… (A–F) Link previewPreview a page's share cards (Open Graph, Twitter Cards) TLS certificatesList a domain's SSL/TLS certificates and their expiry RedirectsFollow a URL's redirect chain (301, 302…) Site down?Check whether a website is down for everyone or just for you User agentDisplay and decode your user agent (browser, OS, device)My IP & network — BGP / RPKI · Subnet · Port check
DNS & domains — DNS lookup · DNS propagation · Whois · Punycode / IDN
Email — Email score · Email check · Email headers · Blacklist · DMARC generator · DKIM key
Privacy & passwords — WebRTC leak · Passwords · Password leak
Dev & encoding — JWT decoder · Unix timestamp · Cron expressions · Base64 · MD5/SHA digests · UUID generator