How to control AI crawlers without blocking Google

I keep this split on leej.uk because I want pages to be searchable, I want to be careful about training, and I want a person who asked an assistant a question to still be able to fetch a page when that is appropriate. The llms.txt guide is about giving tools a map. This one is about who is allowed to request the pages at all.

Crawlers, fetchers and training are not the same thing

Three jobs get lumped together as “AI bots”, and they should not be:

• Search crawlers collect pages so a search engine can index them. Googlebot is the one most UK sites care about. Blocking it, or blocking a generic “Google” token that is actually a search crawler, is how you disappear from Search.
• Training (or collection) crawlers visit in bulk to gather text for model training or for a vendor’s long-lived corpus. Several companies publish a dedicated user-agent for this, and some search vendors also offer a separate token for training (Google has used a token of this kind, distinct from Googlebot). Blocking the training token is not the same as blocking search.
• User-triggered fetches happen when a person asks an assistant a question and the tool retrieves a URL to help answer it, around the time of the request. That is closer to a browser visit than to a training crawl. Some vendors use a different agent for this, or they fetch in ways that look like a normal client.

A policy that says “no AI” without this split will either over-block (and hit Search, or hit a fetch a person asked for) or under-block (and allow a training crawler you thought you had stopped). You do not have to allow training in order to stay in Google. You do not have to block every fetch in order to opt out of training.

I cannot promise that every vendor honours robots.txt, or that they will keep the same token. Treat robots.txt as the baseline contract for well-behaved crawlers, and be honest about its limits.

Start with a clear policy, then pick the mechanism

Write the policy in a sentence you could publish. For a small UK site, a typical honest policy looks like one of these:

• Search engines we already use may crawl. Training crawlers are disallowed. User-triggered fetches may retrieve public pages.
• Everything public may be crawled, including training. (This is a real choice, not a default you have to make accidentally.)
• Only search crawlers we name may crawl; training and unattended collection are disallowed; fetches that need the full page should use the public HTML we already ship.

Once that sentence exists, pick mechanisms in this order:

1. robots.txt, with User-agent tokens you have looked up, not guessed. This is the place well-behaved crawlers look, and it is the place you can change without redeploying a firewall.
2. HTTP headers or X-Robots-Tag, when you need a rule on a specific response, or when a file is not HTML.
3. Firewall or CDN rules, when you must stop a crawler that ignores robots.txt, or you need a rate limit. These are powerful and easy to get wrong.
4. A published usage policy, sometimes called an AI usage policy, plus any TDM / ai.txt signals you want to offer as an extra hint.

Do not start at the firewall because it feels decisive. Start at the policy and robots.txt, then tighten. The AI usage policy builder can help you draft the public wording; the wording should still match the rules you actually ship.

robots.txt rules that name the right agents

robots.txt is matched by user-agent token, not by the word “AI” in a log. A rule for User-agent: * applies to everyone that does not have a more specific group, including Googlebot, unless you have also written a Googlebot group. That is the usual way a “block everyone, we’ll allow friends later” file takes a site out of Search.

A safer pattern is to leave the default open (or as open as you already have it), and to name the training crawlers you are opting out of:

# Example only — tokens go stale. Confirm names in a current directory.
User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Those two names are examples of the kind of token you might use, including a training-specific token used by Google that is not Googlebot. I am not listing a complete set, and I am not promising those strings will be the right ones on the day you read this. Look them up in a current AI user-agent directory, then test the file with the AI crawler robots.txt tester and a general robots.txt tester.

Rules that keep you out of trouble:

• Do not Disallow: / for User-agent: * unless you really mean to ask every well-behaved crawler, including Googlebot, to stay out.
• Do not assume a substring match you have not checked. Tokens are defined by the crawler operator. Read their current documentation as well as a directory.
• Allow your /robots.txt, /sitemap.xml and, if you use them, /llms.txt and /llms-full.txt for the agents you want to honour the policy. A file you have disallowed cannot be read as a source of rules or as a map.
• Path-level rules are allowed. You might permit a marketing site and disallow /app/ or a private preview. Be consistent with noindex and with authentication; robots.txt is not an access-control system.

robots.txt is public. Do not use it to hide a secret URL; use authentication.

HTTP headers and X-Robots-Tag

X-Robots-Tag and the HTML robots meta tag are for indexing and snippet behaviour (and some related directives) on a given response. They are useful when a URL should be crawlable but not indexed, or when you need a directive on a PDF or an image that has no HTML head.

They are a poor general substitute for “don’t train on this site”. Support for AI-specific header directives is uneven, and I will not invent a header name and tell you every vendor honours it. If a vendor documents a header or a meta tag, you can add it; still keep the robots.txt rule, which is the one they are more likely to have implemented.

Useful, well-established uses on a small site:

• noindex on thank-you pages, internal search URLs, and thin parameter variants, served as HTML or as X-Robots-Tag: noindex.
• nofollow only when you mean it; it is not a training opt-out.
• Keeping the header and the HTML tag consistent. Conflicting signals are a good way to spend a week in Search Console.

If you already have a canonical and a sitemap, do not use X-Robots-Tag to paper over a duplicate URL problem; fix the URLs, as in the canonical tags guide in this series.

Firewall and CDN rules, and how they go wrong

A WAF or CDN rule can block a crawler that ignores robots.txt, or cap a rude client, or challenge a junk path. It can also block Googlebot because your “bot score” rule was built for form spam, or because you matched Google in a user-agent and caught something you did not mean, or because you challenged every client without a particular header.

If you go this far:

• Allowlist verified search crawlers, using the operator’s current verification method (for Google, that includes the documented reverse-DNS / IP verification process, not a user-agent string alone). User-agent strings are easy to spoof; do not make “contains Googlebot” your only allow rule, and do not make “looks like a bot” your only block rule.
• Prefer a specific rule for a named, verified training crawler over a general “AI” score, unless your CDN vendor has a maintained, documented category you actually trust and can review.
• Do not put a JavaScript challenge in front of the whole site and assume crawlers will pass it. Many will not, including ones you wanted to allow.
• Log a sample and test after every change: request as a normal browser, as Googlebot (in Search Console’s tester, or via a verified path), and as the training agents you intended to block.

The AI crawler firewall rules tool on this site is a way to draft cautious rules; it is not a substitute for reading what your CDN will actually match, and it will not verify Googlebot for you. I have seen a “block bad bots” preset take a site out of Search until someone looked at the 403s. Treat that class of preset as guilty until tested.

If you use Cloudflare or a similar worker-based host, as I do on this site, keep the allow/deny logic in one obvious place, and deploy it with the same care as a homepage change. A firewall rule is a production change.

Usage policies, TDM and ai.txt

A public usage policy will not, by itself, stop a crawler. It is still worth having, because humans, procurement teams and some tools look for it, and because it forces you to write the sentence I started with.

Alongside a policy page, there are emerging machine-readable hints: TDM (text and data mining) reservation signals, and various ai.txt or similar proposals. Treat these as additional, optional signals, not as a replacement for robots.txt, and not as a guarantee. If you add one, keep it consistent with the robots rules, and do not invent a syntax of your own and expect a crawler to guess it. Link the policy from /llms.txt if you have one, so a tool that only fetches the map can still find your terms.

The AI usage policy builder can produce a starting page. For a UK site, say where you are, what the site is, what is permitted, and that the policy is not a substitute for robots.txt or for the law. This is not legal advice; if you need a contractual or statutory position on mining, talk to a solicitor.

Checking that Google can still get in

Before you congratulate yourself on a training opt-out, prove that Search still works.

• Fetch https://your-domain/robots.txt and read the groups. Is there a User-agent: * / Disallow: / you forgot? Is Googlebot covered by a disallow you meant for someone else?
• Use Google’s robots.txt tester and the URL inspection tools in Search Console on the homepage, a typical article, and a URL you intended to keep out.
• Confirm the sitemap is allowed and submitted, and that the homepage returns 200 to a simple HTTP request without a challenge.
• Watch Search Console for a climb in crawl errors, or a drop in crawled pages, in the days after a firewall change. Those changes often show up before a ranking conversation starts.
• Repeat the check when a CDN preset, a bot-fight product, or a “security” plugin updates itself. The regression is often not your last commit; it is a vendor default.

If you also care about other search engines, test those tokens the same way, one at a time. Do not assume a Google allowlist covers them.

A worked policy for a small UK site

Imagine a small studio site, like this one: a homepage, a projects list, a set of free tools, a handful of articles, and a contact page. The owner wants to stay in Google and other conventional search, to be careful about bulk training, and to allow a fetch when a person has asked a question about a public page.

The implementation I would actually ship:

1. Publish a short policy page that says exactly that, and link it from the footer or the privacy page so a human can find it.
2. Leave robots.txt open for * except where you already disallow admin or preview paths. Add specific Disallow: / groups for the current training-crawler tokens you have chosen from a directory, including a Google training token if you mean to opt out of that and if the current documented token is separate from Googlebot. Test that Googlebot is still allowed.
3. Keep /sitemap.xml and /llms.txt allowed. Do not advertise private URLs in either file.
4. Do not add a CDN “block all bots” preset. If a particular crawler ignores robots.txt and is a problem, add a narrow, reviewed rule, with Googlebot verification in mind, and log it.
5. Recheck the directory and Search Console when you add a section of the site, and when a vendor announces a new agent name.

That is enough for most small sites. You can do more — rate limits, a TDM hint, a more aggressive firewall — but you should not do more until this version is tested and boring.

Key takeaways

• Split training crawlers, search crawlers and user-triggered fetches in the policy, then implement that split. You can stay in Google and still opt out of training.
• Start with robots.txt rules that name current, checked tokens. Do not Disallow: / for * unless you mean to ask everyone, including Googlebot, to leave.
• Headers and X-Robots-Tag are for indexing behaviour on specific responses, not a complete training opt-out.
• Firewall rules are easy to over-block. Never rely on a “block bots” preset, and never verify Google only by a user-agent substring.
• A public usage policy and any ai.txt or TDM signals are extras. They should agree with robots.txt.
• Test Googlebot access, with Search Console, after every change, and whenever agent names or CDN presets change. Use a current directory, not a list from memory.

If you want a second pair of eyes on a robots file or a crawler policy, tell me about it.

Frequently asked questions

Written by Lee Johnson. Published 10 November 2026.