Why Google and AI see different HTML

I design pages on leej.uk so the first response is already enough to read, and I built the Rendered vs Raw HTML Diff because “it looks fine on my laptop” is not a crawler test. What follows is how I compare those views and how I keep them from drifting apart.

Three views of the same URL

It helps to name the three views explicitly:

1. The raw response. The bytes the server or the edge returns for the first request, before any client script runs. This is what curl sees, and what a crawler that does not render sees.
2. The rendered document. The DOM after a browser, or a rendering service, has executed JavaScript, applied CSS, and perhaps fetched extra data. This is what you see in DevTools after the page has settled.
3. The view a particular bot actually got. That may be raw, rendered, partial, or a completely different body produced by a firewall, a cookie wall, a geo rule or a login redirect.

Google Search can render many pages, but that is not the same as a promise that every URL is rendered on every visit, or that the rendered result matches your Chrome window. AI crawlers and fetchers vary even more. Some retrieve a page only when a person asked a question. Some crawl on their own schedule. A number of them, at the time of writing, work from the raw HTML and will not run your client bundle. I will not invent a table of which named bots render, because those behaviours change; you have to test the ones you care about.

If those three views disagree about the title, the main text, the links or the prices, you do not have one page. You have a page that depends on who is asking.

This is the same class of bug as a UI that looks correct in a preview and wrong on a device, which is why I care about SwiftUI previews for design QA in app work, and why I treat raw HTML as the preview that search and AI actually get.

What arrives before any JavaScript runs

The first response is the only view you fully control. It should already contain:

• The title, the canonical URL, robots tags and social meta.
• The H1 and the opening answer, not a loading spinner.
• The main body, or a faithful server-rendered version of it.
• The important internal links, as real <a href> elements, not click handlers on a <div>.
• Structured data, if you use it, in the HTML or as a static script, not only injected later.
• Images that matter for the largest contentful paint, with working src or <picture> markup.

A client-side app that ships an empty <div id="root"> and then fills it in is asking every non-rendering crawler to index an empty shell. That pattern can work if you pre-render or server-render the same content, and if a later render does not replace it with something else. It fails when the first response is a placeholder and the real article exists only after a JavaScript fetch.

View the raw source, not only the Inspected DOM. “View source” or curl -s shows the first body. The Elements panel shows the rendered one. The HTML Page Auditor is a quick way to see whether the raw document already has a single H1, a description and some body text.

CSS also matters here, though it is not “content” in the same sense. If the main text is in the HTML but a script or a stylesheet hides it until a cookie banner is accepted, some clients will never reach a readable state.

Rendering, and who is willing to do it

Rendering is extra work. A client has to download scripts, run them, request data, and wait for the DOM to settle. Browsers do this for people. Search engines may do it in a second wave, after they have already seen the raw HTML. Many AI fetchers, monitors and link-preview bots do not do it at all.

Even when a crawler can render:

• Scripts disallowed in robots.txt will not run. If you block /assets/ or /_next/ for Googlebot, you may be denying it the very files needed to draw the page. Allow the assets a renderer needs, or, better, do not depend on them for the meaning of the page.
• Slow or failing scripts time out. A renderer will not wait forever. A long client-side data call can leave it with the loading state.
• User-only actions never happen. Content that appears only after a click, a scroll to a tab, or a “load more” button is easy for a person to reach and easy for a crawler to miss. If that content should be indexed, give it a real URL or include it in the first response.
• The rendered HTML can still differ from your laptop, because the renderer may have a different viewport, cookie jar, location or date.

Google’s documentation on JavaScript SEO is the place to confirm the current behaviour for Search. I treat rendering as a bonus, never as the only way the main text exists. That habit also helps people who browse with scripts blocked, and it helps the answer-engine structure of the page, which depends on headings and paragraphs being present in the HTML.

Blocked assets, cookies and login walls

A page can look complete in a logged-in Chrome session and empty to a bot for dull reasons:

• CSS or JavaScript blocked by robots.txt, as above.
• A cookie or consent wall that replaces the body until a button is clicked. If the first response for a new client is only the wall, that is the page you have published to crawlers.
• A login redirect. If the canonical article sits behind a session, crawlers that do not sign in will index the sign-in page, or nothing useful. Public content has to be public in the first response.
• Soft login or “continue reading” interstitials that hide the rest of the article in the raw HTML. Some of these still include the text in the source; some do not. Check, do not assume.
• Personalisation that changes the H1 or the offer per visitor. A bot may get a generic version, a UK visitor a priced version, and an empty cache a default. Decide which version is the canonical document and make that the raw response for anonymous requests.

This is closer to state and architecture in SwiftUI than it first appears: if the state that produces the UI is not available to the client, the UI will not appear. For a public web page, the “state” a crawler has is the URL and the first response.

Firewall and CDN rules that swap the body

Content delivery networks and bot-management products can serve a different body to a request they classify as a bot: a JavaScript challenge, a CAPTCHA, a 403, or a slim “please wait” page. Your browser, with a normal client hint profile and a solved challenge, then loads the real article. Googlebot, Bingbot or an AI fetcher may never get past the challenge, or may be allowed by a rule you have not tested since the product changed its defaults.

That is sometimes deliberate and sometimes an accident. A challenge in front of /llms.txt or a blog post you want cited is usually an accident. A challenge in front of /admin is not.

A few cautions:

• Allowlists go stale. User-agent tokens and official IP ranges change. A rule that once let Googlebot through can start serving the challenge page after a product update.
• Cloaking is a policy risk. If you intentionally show search engines a different article body from the one you show people, you are in dangerous territory. A challenge page versus the real page is already a difference; a rewritten “SEO version” is worse. Keep the allowed-bot body and the human body the same document.
• Edge logic can depend on cookies, country or ASN. A crawler may be classified in a way you did not rehearse.

The AI Crawler Firewall Rules tool is a way to think through those allow and deny decisions. The guide to controlling AI crawlers covers the policy side. After any change, fetch the public URL with a plain HTTP client and, if you can, with a verified crawler identity, and read the body you got, not the body you expected.

Sites served from Cloudflare Workers have an extra reason to be careful: the Worker is the origin for that request. There is a separate article on search-friendly Cloudflare Workers sites. The short version is that the first response from the Worker should already be the page.

How to compare raw and rendered HTML

Do not argue from screenshots. Compare the documents.

1. Fetch the raw HTML with curl or an equivalent, from a machine that is not logged in, following redirects, on the canonical host. Save the body.
2. Load the same URL in a clean browser profile, wait for the network to go quiet, and export the DOM or a full HTML snapshot after JavaScript has run.
3. Diff the two. The Rendered vs Raw HTML Diff is built for this. You are looking for text, headings, links, prices and main images that appear in only one side.
4. Repeat a raw fetch with a few user-agent strings you care about, and from outside your office network if a firewall is in play. If the body changes, you have a bot rule or a personalisation rule to inspect.
5. Check robots.txt for the asset paths the rendered version needs, in case a rendering crawler is being told not to load them.

Ignore differences that are purely decorative: a theme class, a debug comment, a CSRF token. Do not ignore a missing H1, an empty article, a “enable JavaScript” message, or a challenge page in the raw view.

When the two views disagree, decide which one is wrong. Almost always the fix is to put the missing content into the first response, not to ask more bots to render.

Designing pages so the first response is enough

The durable design is boring, and it is the one I use:

• Render the article or the tool’s explanatory content on the server, or pre-render it at build time, so the first body is complete.
• Keep client JavaScript for interaction: the theme switch, form behaviour, the tool’s live calculator. The surrounding explanation should not depend on it.
• Use real links and real form actions. Progressive enhancement is not nostalgia; it is how a non-rendering client still understands the page.
• Put the answer in the first paragraph, as I describe in writing for AI answers and search, because that paragraph is often all a fetcher will lift.
• Do not hide the main image or the main text behind a hydration step.
• Treat accessibility and crawling as allied problems. A page that already works without scripts, and that has a sensible heading structure, is easier to use with assistive technology in spirit, even though that article is about iOS, and easier for a crawler to quote.

You do not have to abandon a modern front-end. You have to stop treating an empty shell as a page.

A short debug sequence

When a URL ranks on the wrong snippet, is ignored by an assistant, or “is not being indexed” despite looking fine to you, I run this sequence:

1. curl -sSI the canonical URL. Confirm 200, HTTPS, the expected host, and that you were not redirected to a login or a regional copy you did not intend.
2. Read the raw body. Is the main text there? Is there one H1? Is there a noindex, a canonical to somewhere else, or a challenge page?
3. Diff raw against rendered. Note anything important that appears only after JavaScript.
4. Check robots.txt for the page and its assets, and check for X-Robots-Tag on the response.
5. Fetch again with a bot user-agent, and from a second network if a firewall is possible. Compare bodies.
6. If a CDN dashboard shows a bot score or a challenge, inspect the rule that fired, then fetch again after a change.
7. Only then look at Search Console or an AI crawl log. The HTML disagreement is often enough to explain the symptom.

This is the same spirit as finding performance problems with Instruments: measure the actual artefact, not the one you hoped was shipped.

Key takeaways

• A browser, a search crawler and an AI fetcher can all receive different HTML for one URL. The first response, rendering, blocked assets and firewalls are the usual causes.
• Put the meaning of the page in the initial HTML. Treat rendering as a bonus, not as the only way the article exists.
• Compare raw and rendered documents, and repeat the raw fetch as a bot and from another network, before you trust a visual check in your own browser.
• Do not block the assets a renderer would need, and do not put a challenge page in front of content you want indexed or cited.
• When something looks “fine to you” but fails in search or in an assistant, debug the body each client actually received.

If you want a second pair of eyes on a page that crawlers seem to be reading differently from you, tell me about it.

Frequently asked questions

Written by Lee Johnson. Published 10 November 2026.