How to keep an XML sitemap accurate

I generate the sitemap on leej.uk from the same release as the HTML, and I built the XML Sitemap Validator and the Sitemap Coverage Diff because a file that slowly drifts from the live site is worse than no sitemap at all. What follows is how I keep the two in agreement.

A sitemap is a claim about the site, not a wish list

Google’s own sitemap documentation describes the file as a way to tell search engines which pages you think are important, and to help them discover URLs they might otherwise miss. That is useful on a site with tools, articles and a few quietly linked legal pages. It is not a ranking lever, and it will not force a thin or blocked URL into the index.

The useful habit is to treat the sitemap the same way I treat a release when I maintain an iOS app after launch: ship the list with the thing it describes, and refuse to list work that is not actually available. The case study on small, focused Apple apps makes the same point in a different context. A sitemap that advertises a URL you took down last month is the web equivalent of an App Store screenshot for a feature you removed.

A short, honest file beats a long, decorative one. Search engines already crawl links. The sitemap’s job is to be the clean, complete set of canonical, indexable URLs, with dates you can defend.

What to include, and what to leave out

Include a URL only when all of these are true:

• It returns 200 on the canonical host, over HTTPS.
• It is the canonical version of that page, not a duplicate with a tracking parameter, a trailing-slash variant, or a www twin.
• It is indexable: no noindex in a meta tag or X-Robots-Tag, and not disallowed in robots.txt for the crawlers you care about.
• A person can reach the same content without a login, a paywall that hides the body, or a JavaScript wall that leaves the first response empty.
• You actually want it in search results. Thank-you pages, internal search result URLs, paginated filter combinations and preview deployments usually fail this test.

Leave out, or remove as soon as you find them:

• Redirects (301, 302, 308) and gone URLs (404, 410). Point crawlers at the destination, or at nothing.
• Alternate versions you have already consolidated. The canonical tags guide covers that split; the sitemap should list only the winner.
• URLs that soft-404, meaning they return 200 with “not found” or empty content. Those are a separate mess; the soft 404 guide is the place to fix them, and they should not sit in the sitemap while you do.
• Staging, preview and noindex drafts. If a Worker route still serves a branch preview, that host needs its own robots rules, not a line in the production sitemap.
• Files you do not want indexed as standalone results, unless you have a deliberate reason to list them.

A compact example for a small UK studio site might look like this. The URLs are imaginary; the shape is not:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-10-02</lastmod>
  </url>
  <url>
    <loc>https://example.com/work</loc>
    <lastmod>2026-09-18</lastmod>
  </url>
  <url>
    <loc>https://example.com/articles/how-we-quote-a-build</loc>
    <lastmod>2026-11-04</lastmod>
  </url>
  <url>
    <loc>https://example.com/privacy</loc>
    <lastmod>2026-03-12</lastmod>
  </url>
</urlset>

changefreq and priority are still in the Sitemaps protocol. Google has said it ignores them, and I no longer emit them. If another engine you care about still reads them, keep the values honest; do not set every page to 1.0 and hourly.

The XML Sitemap Generator is a reasonable way to get a first file from a list of URLs. It will not decide what should be on the list. That decision is the whole job.

If the site grows past a single file, use a sitemap index and keep each child under the protocol limits (50,000 URLs or 50 MB uncompressed, as the protocol states). Split by type if that makes coverage diffs easier: pages, articles, tools. Do not split just to look complete.

lastmod that only moves when the page really changed

lastmod is the field most sitemaps lie about. The date should be the last time the meaningful content of that URL changed: the title, the body, a price, a policy clause, a heading structure that would alter a snippet. It should not move because you:

• redeployed the site and the HTML template picked up a new cache-busting query string
• changed the footer, the cookie banner or the theme switch
• rebuilt every page in the static pipeline on one night
• ran a script that stamps now() on every record

Google has been clear that it may use lastmod as a signal when the dates are consistent with reality, and that it may ignore the field when they are not. An always-fresh sitemap trains crawlers to distrust you.

A practical way to set the date:

1. Prefer a content hash or a source-file mtime for that page, not the time the generator ran.
2. If the page is built from a markdown file or a CMS record, use the commit or updated-at of that record.
3. If a shared chrome change really does alter every page in a way that matters to search (a site-wide title suffix, a new canonical host), you can bump every lastmod. That should be rare, and you should be able to say why.
4. Use W3C date format. A date (2026-11-04) is enough for most small sites; add a time and offset only if you genuinely have that precision.

On this site, a new article or tool lands in the sitemap in the same release that publishes the HTML. The lastmod for /articles moves when the index changes; the lastmod for an older guide does not, unless I edited that guide.

Images, videos and news, when they are worth a file

A page sitemap already helps a crawler find the HTML that embeds an image. A separate image or video sitemap is worth the extra work when the media is the thing you want discovered, and when the files are not easily found by crawling the pages alone.

Typical cases:

• A portfolio or product catalogue where each image should be eligible for image search, with its own licence or caption data.
• A library of videos where you can supply a title, description, thumbnail and duration that match the public page.

Skip a dedicated media sitemap when the site is a handful of pages with decorative images, or when you would only be repeating the same hero photograph on every URL. News sitemaps are for publishers in Google News; a studio blog should not file one for the sake of looking thorough.

If you do need the extra file, the Image and Video Sitemap Generator will shape the XML. The contents still have to be true: working image URLs, captions that appear on the page, and no leaked originals from a staging bucket. Google’s image sitemap and video sitemap documents are the rules to follow, and those rules can be updated, so read them when you first add the file.

List the media sitemap in your sitemap index or in robots.txt the same way you list the page sitemap. Do not leave it as an orphan file only you know about.

Finding coverage gaps and leftover URLs

The two failure modes are opposite, and both show up on small sites:

• Gaps. An indexable page exists, it is linked somewhere, and it is missing from the sitemap. New articles, a legal page added by hand, a tool that shipped in a hurry, and a canonical that changed host are the usual absences.
• Leftovers. The sitemap still lists a URL that redirects, 404s, is noindex, or was replaced. Renames and migrations are the usual cause; the website migration SEO checklist is the process for the larger version of this mistake.

A coverage check is a three-way comparison, not a glance at the file:

1. The URLs in the sitemap.
2. The URLs that are actually live and indexable (from a crawl, from your generator’s page list, or from the same manifest you deploy).
3. The URLs search engines already know about, if you have a Search Console export.

The Sitemap Coverage Diff is built for the first two lists. Anything in the sitemap but not live is a leftover. Anything live and indexable but not in the sitemap is a gap. Resolve each one: add, remove, or fix the page.

Search Console’s sitemap and page-indexing reports are the third list. They will not always agree with your file on the same day, because crawling and indexing lag. A URL that you added this morning can be missing from the report without being a bug. A URL you removed last month and that still appears as “crawled but not indexed” with a leftover sitemap entry is a bug.

robots.txt, noindex and the sitemap must agree

Pick one story and tell it everywhere. A URL should not be advertised in the sitemap if you have also:

• disallowed it in robots.txt for Googlebot or the other crawlers you want to use the sitemap
• marked it noindex
• canonicalised it to a different URL

The crawler that honours robots.txt may never see the noindex tag on a disallowed page, and a sitemap entry for that URL just creates a contradiction. If you do not want the page indexed, leave it out of the sitemap and use the right robots mechanism, which is a different article: robots.txt versus X-Robots-Tag.

Mention the sitemap in robots.txt with an absolute URL:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Submit the same URL in Search Console, and in Bing Webmaster Tools if you use Bing. Submitting does not replace a correct file. After a launch, I also ping participating IndexNow engines for new URLs; that is a notification, not a substitute for the sitemap, and it is covered in the IndexNow indexing guide.

Automation that does not invent dates

Once the site has more than a handful of URLs, generate the sitemap in the build, from the same source of truth as the HTML. A hand-edited XML file will fall behind.

The automation should:

• take its URL list from the pages you actually deploy, not from a separate spreadsheet
• set lastmod from the content’s own change time, as above
• refuse to emit a URL that the build has marked noindex, or that is not in the production host list
• fail the release if the file is not well-formed XML, if a loc is relative, or if the file is empty
• write the sitemap into the same release artefact as the pages, so you cannot ship new HTML and yesterday’s map

That is close to the discipline in the iOS app MVP checklist: do not invent a complete-looking artefact that the product cannot stand behind. A generator that stamps today’s date on 200 unchanged articles is inventing a fact.

I validate the output with the XML Sitemap Validator in the same pass, then run a coverage diff against the live URL list. If either check fails, the sitemap does not ship.

A maintenance routine

A sitemap is not a launch-only task. The routine I use, sized for a site of tens or hundreds of pages, is:

• On every content release. Rebuild the sitemap, check that new and removed URLs are reflected, and confirm lastmod only moved on the pages that changed.
• After a rename, redirect or canonical change. Treat leftover sitemap entries as a launch blocker, the same way a missing redirect would be.
• On a monthly calendar reminder. Fetch the live sitemap, run a coverage diff, spot-check a sample of loc values with a real HTTP request, and read the Search Console sitemap report for submitted-versus-indexed gaps you can explain.
• After a migration. Recrawl, rebuild, and keep the old sitemap only if you are still serving the old host and have a deliberate reason to do so. Usually you should not.

A new site should include the same coverage check in its technical SEO launch checklist. This is not a full-time job on a small site. It is a short, repeatable check. If you cannot keep the file current, keep it small and generated, never hand-curated and forgotten.

Key takeaways

• A sitemap should list only live, canonical, indexable URLs you want in search. It is a claim about the site as it is, not as you hope it will be indexed.
• Set lastmod from a real content change, not from the time you deployed the template. Inaccurate dates train crawlers to ignore the field.
• Add image, video or news sitemaps only when the media itself needs discovery, and keep those files as truthful as the page sitemap.
• Find drift with a coverage diff: leftovers in the file, gaps on the site, and Search Console as a slower third view.
• Generate the sitemap in the same release as the HTML, mention it in robots.txt, and re-check it when URLs change.

If you want a second pair of eyes on a sitemap that has drifted, tell me about it.

Frequently asked questions

Written by Lee Johnson. Published 10 November 2026.