robots.txt vs X-Robots-Tag: which to use

I keep these rules tight on leej.uk, and I built testers for them because a single contradictory line is enough to leave a preview URL in an index, or to hide a page you wanted found. The method below is the one I use when I have to choose.

Two different jobs: crawling and indexing

Crawling is the request. Indexing is the decision to store the URL and, usually, its content, so it can appear in search results. Those steps are separate, and they use separate controls.

• Crawl control lives mainly in /robots.txt. It is a set of allow and disallow rules, matched against a user-agent, as described in Google’s robots.txt introduction and in RFC 9309. A well-behaved crawler that is disallowed should not fetch the body.
• Index and snippet control lives on the response: a <meta name="robots"> tag in HTML, or an X-Robots-Tag HTTP header on any content type. Google documents both in the robots meta tag specification. The crawler has to see the response to honour the tag.

A third, weaker effect is worth naming so it does not confuse the choice. Search engines can sometimes list a URL they have not crawled, if they have seen it in a link or a sitemap. They will not have useful content to show, but the URL can still appear. robots.txt does not erase a URL from the web’s memory. noindex does not work on a response the crawler is forbidden to request.

The same split appears in other work I do. A privacy manifest declares what an app may access; it is not the same as the entitlement that actually opens the API. Here, robots.txt is the gate on the request, and the robots tags are the policy on the document.

What robots.txt can and cannot do

A typical file looks like this:

User-agent: *
Disallow: /preview/
Disallow: /search
Allow: /

Sitemap: https://example.com/sitemap.xml

What that can do, when the crawler honours the file:

• Stop listed user-agents from requesting matching paths, which saves crawl budget and keeps thin or private URL patterns out of a routine crawl.
• Point those crawlers at your sitemap, which is a discovery hint, not an override of Disallow.
• Make exceptions with Allow when a broader Disallow would otherwise hide a file you want read, such as /llms.txt under a cautious rule.

What it cannot do:

• It cannot noindex a page. There is no official, portable Noindex directive in robots.txt that you should rely on. Google has said it does not treat such a line as a supported way to remove a URL from the index.
• It cannot reliably keep a URL out of search results if other pages link to it. The URL may still be indexed without a snippet.
• It is not access control. A Disallow is a request, not a lock. Anyone and any bot that ignores the file can still fetch the path. If /preview/ or /admin/ must stay private, use authentication, not robots rules. I treat this the same way I treat a pre-release build in the App Store launch process: do not publish a URL and then hope a text file will hide it.
• It cannot set snippet length, preview images, or archive links. Those are response-level directives.
• It is not a way to block a single PDF or image by content type. You can disallow a path prefix that happens to hold those files, but the mechanism is still the path, not the file format.

Test the file with the real user-agent tokens you care about, not with a guess. The robots.txt Tester on this site is built for that check. Tokens change, so I will not pretend a list here stays complete.

Meta robots on HTML pages

For an HTML page you can ship a tag in the document head:

<meta name="robots" content="noindex, follow">

Common directives, which can be combined:

• index / noindex — whether the page may appear in search results.
• follow / nofollow — whether links on the page may be used for crawling. nofollow is a hint; it is not a promise that no crawler will request the targets.
• noarchive — do not show a cached copy in the results.
• nosnippet, max-snippet, max-image-preview, max-video-preview — control how much of the page may be shown as a preview. The Snippet Control Tester helps you read a proposed combination before you ship it.

You can also target a family of crawlers with name="googlebot" or another token, instead of robots. Unrecognised tokens are ignored by crawlers that do not use them, which is usually what you want.

Two practical rules:

1. The tag must be in the first HTML response, not injected only after JavaScript runs, if you need crawlers that do not render to see it.
2. The page must be allowed in robots.txt for the crawlers that should honour the tag. Otherwise they may never fetch it.

noindex on a thank-you page, a faceted filter, or a thin tag index is a normal, precise use. Hiding a whole section you are ashamed of is usually a content problem, not a robots problem.

X-Robots-Tag for files, PDFs and anything without a head

X-Robots-Tag is the same vocabulary, sent as an HTTP header. It is the right tool whenever there is no HTML <head> to put a meta tag in, or when you want one rule at the server for a class of responses.

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, noarchive

You can also name a bot:

X-Robots-Tag: googlebot: noindex
X-Robots-Tag: otherbot: nofollow

Use it for:

• PDFs, office documents and images that are useful to people who have the link, but should not appear as standalone search results.
• API responses or downloads where a meta tag is impossible.
• A site-wide default applied in a Worker or origin, with more specific rules on particular paths.

The header must be present on the response the crawler actually receives, after redirects. A noindex on the redirecting URL is not a substitute for a noindex on the destination, or for leaving the destination out of the sitemap. The X-Robots-Tag Rule Builder is a way to draft the header; you still have to attach it at the origin or the edge and then confirm with a real request, for example via the HTTP Header Inspector if you are already checking headers.

If both a meta tag and an X-Robots-Tag are present, crawlers that support both will generally apply the strictest combination. Do not set them to opposite meanings and hope.

A simple rule for choosing

I use this, in order:

1. If the URL must not be fetched by strangers, do not publish it. Use authentication, network controls, or do not put the document on a public host. robots.txt is not the control.
2. If a crawler should not spend time on a path, and you do not need it to read a noindex tag there, disallow the path in robots.txt. Good candidates: internal search result URLs, infinite filter combinations, previews already blocked by noindex and removed from the index, or known duplicate parameter patterns.
3. If the document should be fetchable but not indexed, allow the crawl and send noindex — a meta tag on HTML, X-Robots-Tag on everything else.
4. If you only want to limit the snippet or the cached copy, allow the crawl and use snippet directives, not Disallow.
5. If you want the page in search, do not disallow it, do not noindex it, and list the canonical URL in the sitemap if you keep one.

When I add a new section, I decide this before the first deploy, in the same spirit as maintaining an iOS app after launch: the policy ships with the URL, not as a cleanup ticket after a crawler has already found it.

How they interact, including the noindex-and-disallow trap

The trap is this: you want a URL out of the index, so you add <meta name="robots" content="noindex"> and, to be “safe”, you also add Disallow: /that-page to robots.txt. A crawler that honours robots.txt now cannot fetch the page, so it may never see the noindex. If the URL was already indexed, or is discovered from a link, it can stay in the results as a bare URL. You have blocked the only instruction that would have removed it.

The safer sequence for a URL you want deindexed:

1. Allow the path in robots.txt.
2. Serve noindex (meta or X-Robots-Tag) on the live response.
3. Remove the URL from the sitemap.
4. Wait until the URL is reported as excluded by noindex in Search Console, or until you have given crawlers reasonable time to refetch it.
5. Only then, if you still want to save crawl budget, add a Disallow. Understand that a later policy change will be slower, because disallowed paths are fetched less often.

Google Search Console also offers a temporary removal tool. That is a short-term hide, not a substitute for noindex, and it does not apply to every search engine.

Other interactions to watch:

• A sitemap entry for a disallowed or noindex URL is a contradiction. Drop it from the sitemap.
• nofollow on a page does not noindex that page. It only talks about the links it contains.
• A canonical tag points to the preferred URL; it is not a noindex. Use canonicals for duplicates, and noindex for pages that should not appear at all.

AI crawlers and the same distinction

The same split applies to AI crawlers and fetchers, with one extra policy question: some bots crawl to build a training corpus, some fetch a page only when a person asked a question, and some do both. Tokens and behaviour change, so I will not invent a permanent list here.

What does not change:

• robots.txt is still the place to allow or disallow a named user-agent from requesting paths, if that crawler honours the file. The guide to controlling AI crawlers is the practical walkthrough.
• A usage policy, or a file such as llms.txt, is not a crawl or index control. How to write llms.txt is about giving assistants an honest map, not about blocking them.
• If you serve a noindex or an X-Robots-Tag for a particular bot, that bot still has to be allowed to fetch the response.

The robots.txt and llms.txt Generator will help you keep those files consistent. Test the user-agent strings you actually care about, and review the rules when you change policy, rather than pasting a block from a thread and assuming it still matches the crawler names.

Worked examples

A booking confirmation page. People need to load it. You do not want it in search. Allow it in robots.txt. Add <meta name="robots" content="noindex, follow">. Leave it out of the sitemap. Do not disallow it until, and unless, it has dropped out of the index and you have a reason to stop crawlers requesting it.

A PDF price list at /downloads/prices.pdf. There is no HTML head. Send X-Robots-Tag: noindex on the PDF response. Link it from an indexable page if customers need it. Do not rely on Disallow: /downloads/ if you also want a crawler to see the header, and do not put the PDF in the sitemap.

Preview deployments at preview.example.com. This is not a robots-tag problem first. Password the host, or keep it off the public internet. Add Disallow: / in that host’s robots.txt and a site-wide noindex as defence in depth, never as the only lock.

An image that should be visible on an article but not as its own image-search result. You cannot achieve that with a page-level meta tag alone. Send X-Robots-Tag: noindex on the image response, or keep the file behind a path policy you are willing to live with. Check the response headers, not only the article HTML.

A section you plan to launch next week. Do not list it in the sitemap. A noindex until launch is reasonable if the URLs are already public. Removing the noindex is part of the launch checklist, not a later tidy-up.

Key takeaways

• robots.txt governs crawling: whether a bot may request a URL. Meta robots and X-Robots-Tag govern indexing and snippets on a response the bot is allowed to fetch.
• You cannot reliably noindex a page from robots.txt. Disallowing a URL that already has noindex can prevent the crawler from seeing the tag.
• Use a meta robots tag on HTML, and X-Robots-Tag on PDFs, images and anything without a head.
• If a URL must stay private, authenticate it. Robots rules are not an access control.
• Apply the same crawl-versus-index split to AI crawlers, test real user-agent tokens, and keep the sitemap in agreement with the rules you publish.

If you want a second pair of eyes on a robots setup that is fighting itself, tell me about it.

Frequently asked questions

Written by Lee Johnson. Published 10 November 2026.