Canonical tags, trailing slashes and duplicate URLs

I have had to be strict about this on leej.uk, which is a small, static-ish site served from a Cloudflare Worker: one host, HTTPS, a consistent path shape, and a canonical on every public page. The method below is a URL policy you can write down, not a tour of a single tool. The canonical conflict finder and the URL normalisation tester are there to check the policy once it exists.

What ‘duplicate’ actually means to a crawler

Search engines try to group addresses that show the same content, and to pick one as the canonical. You can help or you can get in the way. Duplicates are not only copy-pasted blog posts; they are, much more often on a small site:

• the same page at two technically different URLs,
• a URL that redirects, eventually, to another that is already in the sitemap,
• a tracking parameter that creates an infinite family of addresses for one document.

Google has said that they use several signals — redirects, the rel=canonical link, sitemaps, internal links — and that they may ignore a canonical you set if it is inconsistent or implausible. I treat that as a reason to be boring and consistent, not as a reason to try to “force” a URL with five conflicting signals.

Duplicates waste crawl budget (less of a problem on a fifty-page site than on a faceted shop, but still messy), they split signals, and they make it easy to share the wrong URL. They also confuse any assistant that is trying to cite “the” address of a page, which is why the same shape should appear in your sitemap, your llms.txt, and your internal links.

A page that is similar but genuinely aimed at a different user (a UK and a US version, a print view with different content) is not a duplicate you should canonicalise away without thought. A page that is the same article with a trailing slash is.

The usual sources: scheme, host, slash, case and parameters

Five families cause almost all of the trouble:

Scheme. The same path on HTTP and on HTTPS, for example example.com/about with each scheme. The redirect should be a single hop to HTTPS. HSTS is a complement, not a substitute for the redirect, because the first visit and non-browser clients still need a clear 301.

Host. www.example.com and example.com. Pick one. Either is fine; mixing them is not. The certificate, the DNS, the sitemap and the canonical all have to use the same host. A site that “works” on both, with no redirect, is two sites.

Trailing slash. /tools and /tools/ are different URLs. Servers and frameworks disagree about which is natural. Pick one policy for directory-like paths, and another, if you must, for files that look like sitemap.xml. Then redirect the other shape. Do not serve 200 on both with a canonical as the only hint if you can avoid it; the redirect is the stronger, clearer signal.

Case. /About and /about. Many servers are case-sensitive; some are not. Humans will type either. A lower-case policy, with a redirect from mixed case, is the usual choice for a public site that does not have a reason to be case-sensitive. When you mint a new path, use a deliberate slug, not a heading pasted with spaces and capitals.

Parameters. ?utm_source=, ?ref=, session IDs, sort order, ?page=1 on a URL that is already the first page, print views. Some parameters change the content; most marketing ones should not. The canonical of a purely tracked URL is the clean URL. The clean URL should not redirect in a loop when the parameter is present; it should 200, with a canonical (and, ideally, a parameter you do not need to index).

A sixth, quieter one: default documents, such as /about and /about/index.html, or a preview host that also serves the production content. If a second host must exist (a preview, a staging site), it should be noindex, preferably blocked from production search via robots or authentication, and it should not appear in the production sitemap.

Pick one URL shape and stick to it

Write the policy in one short block you can hand to a future you, or to a host configuration:

• HTTPS only.
• Bare host, or www, but not both. (On this site: the bare host.)
• Lower-case paths.
• Trailing slashes: yes or no, with explicit exceptions for files.
• Query parameters that are part of the resource (a search query on a search page, a documented API filter) versus parameters that are stripped from the canonical.

Every public URL you publish — in links, emails, sitemaps, llms.txt, App Store text, QR codes — should be in that shape. Internal links should never point at a URL you intend to redirect; they should point at the destination. That saves a hop and keeps crawlers from discovering the weak form first.

If you are building routes, generate them from one helper, not from a mix of hard-coded strings. The URL normalisation tester is a useful way to see how a candidate URL collapses under your rules before you ship it.

Changing the policy later is a migration. Treat it like one, with a map and 301s, as in the website migration SEO checklist. Do not flip the trailing-slash rule on a Tuesday because a new framework defaulted the other way.

Redirects, then the canonical tag, in that order

A redirect says “this address is not the resource; go there”. A canonical says “this address is the resource, and that other address is the preferred one if you somehow see both”. Use them in that order.

• Redirect any URL that is not the chosen shape: scheme, host, slash, case, and obsolete paths from a redesign. Prefer a single 301 (or 308, when you mean to keep the method) to the final URL. Avoid chains: HTTP on the www host, then HTTPS on www, then the bare HTTPS host is two hops you can collapse. The redirect chain tracer will show you the chain; you should not need more than one hop for a normalisation.
• Then serve the chosen URL with a 200 and a self-referencing canonical that is exactly that URL, including scheme, host, path and slash.

<link rel="canonical" href="https://example.com/tools/slug-generator">

Absolute HTTPS URLs in the canonical are the least ambiguous. A relative canonical is easier to get wrong when the page is served on more than one host. The href should be the URL you want in Search, not a parameterised version of itself, and not a typo.

Do not canonicalise a live page to a URL that 404s, that redirects again, or that you have noindexed, unless you have a very specific reason and you understand the conflict you are creating. Do not point every page at the homepage as a “hack”. That is not a canonical policy; it is a request to drop the pages.

Self-referencing canonicals, and when not to point away

Self-referencing canonicals look redundant, and they are useful. They tell a crawler that, even if it found this page with a tracking parameter or via a second host you failed to redirect, the preferred address is the one you named. They also make it obvious, in the HTML, which URL you think this is.

Point the canonical away only when the page you are serving is a true duplicate or a variant you do not want chosen: a printer-friendly copy, a syndicated copy you are allowed to publish but not to have indexed as original, a filtered list that is a subset of a canonical list. In those cases the canonical must be a 200 URL you control, or a URL the receiving site has agreed to, and the rest of your signals (sitemap, internal links) should agree.

Cross-domain canonicals exist, but they are a syndication tool, not a way to merge two websites you accidentally launched. If you are moving a site, redirects are the primary signal, as in the migration checklist; a canonical-only move is weaker and slower.

If you cannot decide whether two URLs are duplicates, do not point one at the other “just in case”. You may accidentally drop a useful, slightly different page. Decide, or leave them as two pages with their own self-canonicals and their own titles.

Pagination, filters and parameters

List pages are where small sites grow accidental duplicates.

• Pagination. If you paginate, each page should have a unique URL, its own self-canonical, and a unique enough title or heading that page 2 is not a clone of page 1 in the first screen of text. rel=prev/next is historically messy as a ranking signal; I would not rely on it. Do not canonicalise page 2 to page 1 if page 2 has different items; that asks a crawler to pretend those items live on page 1.
• Filters and sort orders. If a filter makes a genuinely useful, indexable landing page, give it a clean URL and unique introductory text. If it is a tool for users already on the list, keep it as a parameter, self-canonical to the unfiltered list, or noindex it. Infinite combinations of facets are a soft 404 factory as well as a duplicate factory.
• ?page=1 and empty filters should redirect to, or canonicalise to, the clean list URL.
• Tracking parameters should not appear in sitemaps or in internal links. The page may still 200 when they are present.

If you are unsure whether a parameterised URL is being indexed, look in Search Console for the naked and the parameterised forms, and look at the canonical the live HTML declares. The conflict finder is built for that comparison.

How to find conflicts on a real site

A method that scales to a few hundred URLs, which is most of the sites I work on:

1. List the shapes you think you support. Bare and www, slash and no slash, a sample parameter, HTTP if it still answers.
2. Request them, and record the status, the final URL after redirects, and the canonical in the final HTML. The redirect chain tracer and the canonical conflict finder will do the mechanical part.
3. Diff the sitemap against that list. Every sitemap URL should be a 200, in the chosen shape, self-canonical, and not a redirect. Every important 200 should be in the sitemap, or left out on purpose.
4. Grep the HTML for internal links that still use the weak shape, including an HTTP scheme and the wrong host. Fix the links; do not rely on redirects for your own site.
5. Check Search Console (the Page indexing and URL inspection reports, and a site: search if you must) for unexpected variants that are indexed or “crawled, not currently indexed”.

On this site I also refuse a deploy that would ship a page without a canonical, or with an ops host in a public link. You may not need that level of automation, but you do need the check to happen before a launch, not after a week of “why is this URL odd in the results”.

A short policy you can write down

Here is a complete, boring policy a small HTTPS site can adopt. Change the host and the slash rule to match your choice, then actually implement it.

• Published URLs are https://example.com/path, lower case, no trailing slash except for the root /. (Or: trailing slashes on all directory-like paths. Pick one.)
• http, www, mixed case, and a trailing slash on a no-slash site 301 in one hop to the published URL.
• Every public HTML page includes <link rel="canonical" href="…"> pointing at its published URL.
• Sitemaps, internal links, llms.txt and share meta tags use only published URLs.
• Tracking parameters are allowed on the request, are not used in links we write, and do not change the canonical.
• Staging hosts are noindex and are not in the production sitemap.
• A change to this policy is a migration, with a URL map.

If a new tool or framework wants a different default, the policy wins, or the policy is formally changed. The framework does not get to quietly serve both.

Key takeaways

• Duplicates on a small site are usually scheme, host, slash, case or parameters, not copied essays.
• Pick one URL shape and use it in links, sitemaps and canonicals. Redirect the other shapes in a single hop.
• Put a self-referencing absolute canonical on the 200 URL you kept. Do not point canonicals at redirects, 404s or the homepage as a trick.
• Pagination and useful filters should be distinct pages; tracking parameters and empty filters should not.
• Find conflicts by requesting the variant shapes, checking the sitemap, grepping internal links, and using Search Console. Tools can do the fetching; you still have to decide the policy.

If you want a second pair of eyes on a URL policy or a conflict report, tell me about it.

Frequently asked questions

Written by Lee Johnson. Published 10 November 2026.