Paste a robots.txt and a list of URLs and see, for Google, Bing and the main AI crawlers, which URLs are allowed or blocked and the exact rule that decides.
1 robots.txt 2 processed on your device 3 Allowed or blocked
robots.txt Tester
robots.txt
What to test
User agents
Rules
First rules
Results
robots.txt controls crawling, not indexing: a blocked URL can still appear in search if other pages link to it.
01
What it does
Tests URLs against a robots.txt for many crawlers, AI bots included, using RFC 9309 matching, and shows the deciding rule.
RFC 9309 matching: longest rule wins, Allow wins ties
* and $ wildcards
Googlebot, Bingbot and the main AI crawlers at once
Any custom user agent
02
Private by design
Everything happens in your browser. Nothing you type, paste or upload is sent anywhere, and it keeps working offline once the page has loaded.
03
What you get
An allow and block grid, with a Copy and a Download button on every result.
robots.txt
Crawlers
AI bots
SEO
Why it matters
A single line in robots.txt can hide a whole site from search, or let AI crawlers into areas you meant to keep out. The matching rules aren't obvious: the longest matching rule wins, Allow beats Disallow on a tie, and a crawler uses only the most specific group that names it.
This tester applies the standard (RFC 9309) to every URL for every crawler at once, and shows which rule decided each result. It's the companion to the robots.txt and llms.txt Generator: write the file there, test it here.
How it works
InputA robots.txt and some URLs
In your browserMatched with the RFC 9309 rules
OutputWhich crawler can fetch which URL, and why
Where it fits in a designer's workflow
Plan
Design
Build
Test
Launch
Before blocking or opening a site to crawlers.
By the numbers
Worked out by this tool with the RFC 9309 rules (longest match wins, Allow wins a tie). Each crawler uses only the most specific group naming it, so a Googlebot group replaces the * rules rather than adding to them. Source: RFC 9309: robots.txt.Show the numbers as a table
Crawler
Allowed URLs
Googlebot
7
Bingbot
4
GPTBot
0
OAI-SearchBot
4
ChatGPT-User
4
ClaudeBot
4
Claude-User
4
PerplexityBot
4
Google-Extended
0
Applebot-Extended
4
CCBot
0
Bytespider
4
meta-externalagent
4
Amazonbot
4
How to use it
Paste your robots.txt.
List the URLs or paths you care about.
Read the grid of crawlers and URLs.
Each cell shows the rule that decided it.
FAQ
Does robots.txt stop a page being indexed?
No. It stops crawling. A blocked URL can still appear in results if other sites link to it. Use a noindex tag on a crawlable page to remove it.
Which group does a crawler use?
The group whose User-agent line matches its name most specifically. If none matches, the * group. Groups aren't combined.
Do AI crawlers obey robots.txt?
The main ones from OpenAI, Anthropic, Google, Perplexity and Apple say they do. Some fetchers acting for a user may not.
I build fast, accessible websites and web apps that pass these checks from day one. Tell me about your project and I'll reply personally with honest next steps.
