AI Crawler robots.txt Tester: GPTBot & More

Your robots.txt

Site address

Fetch sends this address to leej.uk, which reads /robots.txt once as LeejTools/1.0 and keeps only a count. Or paste below: pasted text never leaves your device.

robots.txt

Paths to test (one per line)

By kind of crawler

Purpose	Crawlers	Blocked everywhere	Partly blocked	Allowed everywhere

Every crawler, every path

Crawler	Purpose	Group used	/

Rewrite for a policy

AI crawler policy

Edited robots.txt

Want the AI crawler view of your whole site? Run PingAI, which scores how well ChatGPT, Claude, Perplexity, Gemini and Copilot can reach and read it.

What it does

Runs your robots.txt for 35 AI and search crawlers over every path you list, shows the rule that decides each one, and flags the mistakes that block search engines by accident.

35 crawler tokens: AI training, AI search, user fetches, opt-out tokens and classic search engines
The RFC 9309 rules: group matching, the * fallback, wildcards, $, longest match and Allow winning ties
The rule that decides every cell, with its line number
Typos in crawler names, retired tokens and * rules that block search engines too

Private by design

Everything you paste or upload stays in your browser. Only when you press Fetch does leej.uk read the address you give, once, as LeejTools/1.0, keeping nothing but a daily count.

What you get

A crawler by path matrix, a summary by kind of crawler, warnings and a rewritten robots.txt, with a Copy and a Download button on every result.

robots.txt
AI crawlers
GPTBot
ClaudeBot

Why it matters

Most robots.txt files now carry a block of AI crawler rules, often pasted from a blog post. They're easy to get wrong in ways that don't show. A crawler that has its own group ignores the * group completely, so allowing a bot can quietly let it into your checkout. A typo like ClaudeBott blocks nothing. A Disallow added under * to stop AI training also stops Googlebot. And tokens like Google-Extended don't block any crawling at all: they only control how Google uses what Googlebot already fetched.

This tester applies the RFC 9309 rules that Google and the major AI companies follow, for 35 crawler tokens across five kinds of crawler, over the paths you choose. Each cell shows whether the crawler may fetch the path and the exact rule that decided it. The warnings pick out typos, retired tokens and search engines blocked by accident. Choose a policy, such as blocking AI training but allowing AI search, and it rewrites only the AI part of the file, carrying your general rules into each new group so nothing else changes.

How to use it

Fetch your robots.txt, or paste it.
List the paths you care about.
Read the matrix and the warnings.
Pick a policy if you want a rewrite, and download the new file.

FAQ

Does blocking Google-Extended stop Googlebot?

No. Google-Extended is a usage token: it controls whether Gemini can train on or ground answers in your pages. Googlebot still crawls them for Search, and AI Overviews are part of Search.

If I block GPTBot, can ChatGPT still show my pages?

Yes, through OAI-SearchBot and ChatGPT-User, which OpenAI runs separately. Block GPTBot to stop training and keep the others to stay visible in ChatGPT search.

Why can an allowed AI crawler reach my checkout?

A crawler with its own group ignores the * group. If you give it Allow: / alone, it loses your * disallows. The rewrite copies them into each group.

Do AI crawlers obey robots.txt?

The big ones say they do, and their published IP ranges let you check. Some smaller crawlers don't. Use the AI crawler IP verifier and your logs to see who's really visiting.

Work with me

Want a site that gets these right?

I build fast, accessible websites and web apps that pass these checks from day one. Tell me about your project and I'll reply personally with honest next steps.

Start a project What I do