Making a Cloudflare Workers site searchable

This is how I ship leej.uk. The pages are generated ahead of time, the Worker serves the HTML, and the same release includes the sitemap, robots.txt and the AI-facing index files. What follows is the method, not a general Cloudflare tutorial.

Workers can be excellent for search, if the HTML is real

A Worker is just a program that sits on a request and returns a response. That can be a complete HTML document, a redirect, a JSON file, or a cached copy of any of those. Search does not care that the program ran at the edge rather than on a VM, as long as the response looks like a normal website: a stable URL, a 200 for a real page, a 404 for a missing one, and the content in the HTML.

The usual ways a Worker site hides itself are not exotic:

• The HTML in the first response is a loading spinner, and the article arrives later through client-side routing.
• Preview hostnames (*.workers.dev, a staging subdomain) are indexable, so Google has two copies of the site.
• Trailing slashes, www and HTTP all resolve, but not to one canonical URL.
• A long cache on HTML means a typo or a new article is invisible for days, which is the web equivalent of chasing a stale build when you are finding performance problems with Instruments: look at what is actually being served, not at what you last deployed.
• Bot Fight, a WAF rule or a challenge page treats Googlebot or an AI fetcher as an attacker.

If you avoid those, a Worker is a good place to host a content site. You control the headers, the redirects and the body in one place, and you can deploy an article and its sitemap entry together. That is the same discipline I use when I maintain an iOS app after launch, and when I take a build to the App Store: the thing people see and the metadata that describes it should ship in the same release.

Generate HTML on the server, not only in the browser

Crawlers start with the bytes in the HTTP response. Google can also render JavaScript, but rendering is delayed, more expensive, and easy to get wrong; other search engines and many AI tools will not run your front-end at all. The safe default is that the title, the headings, the article text, the internal links and the canonical tag are all present in the raw HTML.

A Worker can produce that HTML in two honest ways:

• Static files in the release, with the Worker choosing the file from the path. That is what this site does for articles, tools and the other public pages.
• HTML built in the Worker from a template and a data store, still returned as text/html in the first response.

A single-page app that boots in the browser is the risky option. If you already have one, prerender the public pages at build time, or have the Worker render them, so a client without JavaScript still gets the words. Then check the difference between the raw response and the rendered DOM. The Rendered vs Raw HTML Diff is for that comparison: if the article text only appears after rendering, crawlers that never run your script will not see it.

A few details are easy to miss:

• Use real <a href> links to other pages. A click handler that calls history.pushState without an href is not a link, and it will not be followed.
• Return a real 404 for unknown paths, with a 404 status and a short HTML body. Do not serve the homepage or an app shell with status 200 for every URL. That is how you create soft 404s.
• Don’t require a challenge or a login to read a public article. If the first response is a JavaScript wall, the page is not public to a crawler.
• Keep a canonical URL in the document head, and make it an absolute https:// URL on your chosen host.

Relative URLs in your own scripts should be resolved against the site origin, not guessed. A small but sharp foot-gun is calling new URL('/path') without a base; in a Worker or a browser that can throw or resolve against the wrong origin. Prefer new URL('/path', 'https://example.com') or, in the browser, new URL('/path', location.origin).

URL rules: host, slash, redirects and canonicals

Pick one hostname, one protocol and one trailing-slash policy, and make every other variant a single redirect to that URL. On this site the canonical host is leej.uk over HTTPS. Whatever you choose, be boring and consistent.

A Worker is a good place to enforce that, because you see the request URL before anything else runs. A typical pattern:

1. If the protocol is http, redirect 301 to https.
2. If the host is www.example.com (or the other way around), redirect 301 to the canonical host, preserving the path and query you intend to keep.
3. If the path is missing or has a surplus trailing slash, redirect 301 to the canonical path.
4. Only then look up the page.

Do not chain these as three hops if you can avoid it. One redirect is enough, and crawlers will follow only so many. The Location header should be an absolute URL.

The canonical tag should then repeat the URL you redirected to, not the URL the request arrived on. If a tracking query is present, decide whether to strip it in the redirect or leave it on the request and omit it from the canonical. Either way, do not mint a second indexable URL for ?utm_source=….

Preview and developer hostnames are a special case, covered later: they should not redirect to production on their own (that surprises you while you are testing), but they must not be indexed.

Cache HTML carefully

Workers sit in front of a very effective cache. That is useful for a CSS file with a content hash in the name, and dangerous for an HTML document that you might edit this afternoon.

A conservative split:

• HTML: a short Cache-Control max-age, or no-cache with an ETag so the browser revalidates. Purge or bypass on deploy so a typo fix is not stuck at the edge.
• Hashed assets (app.3f8c1a.css, images with a version query you actually change): a long max-age is fine, because the URL changes when the file does.
• robots.txt, sitemap, llms.txt: cache them, but not for weeks. A stale sitemap that omits a new article, or still lists a removed one, causes real confusion.

The Cache-Control Explainer will decode a header before you ship it. Two mistakes I watch for:

• Caching a 404 or a 301 for a long time. If you add a page at a path that used to 404, the edge may keep the 404. Use a short TTL on error responses, or purge the path when you add the page.
• Caching HTML with a cookie or a personalised fragment under the same key as the public page. Public article URLs should return the public document.

If you use the Cache API inside the Worker, put a version key in the cache key and bump it on deploy, or delete the entries you have made stale. “It is at the edge” is not a reason to serve last week’s heading. A service worker in the browser is a separate cache again; don’t debug one by changing the other.

Headers, sitemaps and robots.txt from the same origin

Search and AI tools look for a handful of well-known files on the canonical host, and they look at the headers on each page. Serve them from the same origin as the HTML, in the same release.

Each public page should have:

• a unique <title> and meta description
• a canonical URL
• a robots meta tag or HTTP header that allows indexing, unless the page should stay out of search
• Open Graph tags if you care about link previews
• absolute URLs in those tags, on the canonical host

The site as a whole should have:

• /robots.txt, allowing the crawlers you want and pointing at the sitemap
• /sitemap.xml, listing only indexable canonical URLs
• optionally /llms.txt, if you maintain one

The Indexability Checker is a useful pass over a URL: status, canonical, robots and the obvious contradictions. The contradiction I see most on Worker sites is a sitemap entry for a URL that then noindexes, or a robots.txt Disallow that hides a path still listed in the sitemap.

Generate the sitemap from the same list of pages the Worker can actually serve. If a page is added by dropping a file into the release, the sitemap should be built from that list, not from a second, handwritten index that will drift.

Bot fight, WAF and the crawlers you still want in

Cloudflare’s Bot Fight mode, WAF custom rules, rate limits and challenge pages are there to stop abuse. They will also stop a crawler if you are not specific. A blocked Googlebot cannot invent a page from a title tag it never received, and an AI tool that honours a challenge page will see the challenge, not the article.

The principle is: block the traffic you mean to block, and let the rest through with ordinary rate limits. I will not give you a ready-made attack configuration, and I will not pretend a particular product toggle is always safe. What I do on a small content site is narrower:

• Review which managed rules and bot features are on, and test a fetch that uses a documented crawler user-agent, from a published crawler IP range where Cloudflare or the search engine publishes one.
• If you challenge anonymous browsers, confirm that the well-known search crawlers you care about are skipped. Don’t assume they are.
• If you write WAF rules for AI crawlers, be explicit about the user-agent tokens, and keep a copy of the policy you wanted: some fetchers are invited, some are not. The AI Crawler Firewall Rule Builder is a way to draft those rules carefully, not a substitute for testing them against a real request.

User-agent tokens and IP lists change. Revisit the rules when you change policy, and after a Cloudflare or crawler announcement, rather than treating a screenshot of the dashboard as permanent.

If a page is public, the first response should be the page. A JavaScript challenge in front of every URL is a decision to leave some crawlers out. That can be a reasonable choice for a private app; it is usually the wrong choice for an article you want found.

Preview hostnames that must stay noindex

Workers give you preview URLs for free: *.workers.dev, environment subdomains, and any preview. host you add yourself. Those URLs are useful, and they are toxic if Google indexes them.

Treat every non-canonical hostname as noindex, nofollow, and keep them out of the sitemap. A Worker can do this by host:

const CANONICAL_HOST = "leej.uk";

function isPreview(url) {
  return url.hostname !== CANONICAL_HOST;
}

function headersFor(url, base) {
  const headers = new Headers(base);
  if (isPreview(url)) {
    headers.set("X-Robots-Tag", "noindex, nofollow");
  }
  return headers;
}

Also:

• Don’t list preview URLs in the public sitemap, in llms.txt, or in internal links on the production site.
• If a preview host is accidentally crawled, keep the X-Robots-Tag in place and, if you use Search Console, use the removals tools as a temporary help. The durable fix is the header and the missing sitemap entry.
• Basic auth or an IP allow-list on staging is even clearer, if you do not need anyone unauthenticated to see it.

The same rule applies to the default workers.dev route if the project still has one. If you do not need it, disable the public route. If you need it for yourself, noindex it.

A launch and debug checklist

Before you point a domain at a Worker, and again after a release that changes routing or headers:

• Fetch a page with curl -sSI and read the status, content-type, cache-control, canonical-related headers and any x-robots-tag. Then fetch the body and confirm the article text is in the first response.
• Compare raw HTML with a rendered view for a sample of pages, including the homepage, an article and a 404. Use the Rendered vs Raw HTML Diff if you want a structured comparison.
• Follow HTTP, www and trailing-slash variants and confirm a single hop to the canonical URL.
• Request a path that should not exist and confirm a 404 status, not a 200 shell.
• Open /robots.txt and /sitemap.xml on the canonical host. Every sitemap URL should be canonical, indexable and a 200.
• Confirm preview hostnames send noindex and are absent from the sitemap.
• Check a sample of pages in the Indexability Checker.
• Review Bot Fight, WAF and challenge settings against the crawlers you still want, and adjust with care. The AI Crawler Firewall Rule Builder is a drafting aid, not a sign-off.
• After deploy, fetch a page you just changed and confirm you are not still seeing the cached copy.

If something looks wrong in Search Console, start with the live URL inspection and the raw HTML, not with a rewrite of the whole Worker. Most failures I have had to fix were a stale cache, a preview host, or a missing trailing-slash redirect — the boring parts, not the compute.

Key takeaways

• A Worker site ranks when the first response is real HTML, with stable canonical URLs and honest status codes.
• Do the routing in one place: HTTPS, host, trailing slash, then the page, with a single redirect.
• Cache hashed assets for a long time; cache HTML and well-known files cautiously, and purge on deploy.
• Serve robots.txt, the sitemap and the pages from the same origin, and keep them consistent.
• Keep preview hostnames noindex, and review bot and WAF rules so the crawlers you invited can still read the site.

If you want a second pair of eyes on a Worker site, tell me about it.

Frequently asked questions

Written by Lee Johnson. Published 10 November 2026.