Why it matters
A crawl trap is a set of URLs with no end: a calendar that links to next month forever, filters that combine into millions of pages, a session ID that makes every visit a new URL, or a relative link that keeps adding the same folder. Crawlers spend their time there instead of on the pages you want found, and Search Console fills up with "Discovered - currently not indexed".
This finder looks at the URLs themselves. It classifies every parameter, works out how many variants the filter and sort options can produce, turns paths into shapes (/events/{yyyy}/{mm}/{dd}/) to spot calendars, looks for repeated path segments and duplicate spellings, and flags pagination that runs into the hundreds. Each pattern gets a risk level, an example and either a robots.txt rule using Google's wildcard syntax or the fix at the source, which is usually better.
How to use it
- Export the URLs from your crawler, or paste lines from your access log.
- Read the crawl traps, most serious first.
- Check the parameter table for anything you don't recognise.
- Copy the robots.txt rules you agree with and test them before they go live.
FAQ
Should I block crawl traps with robots.txt?
Sometimes. Robots.txt stops crawling, which saves crawl budget, but Google can't then see a canonical or noindex on those URLs. Fixing the source (no links to multi-filter pages, sessions in cookies) is better; robots.txt is the backstop.
What does the wildcard rule /*?*=*&*= do?
It matches any URL with two or more parameters, which is a common way to let one-filter pages be crawled while blocking combinations. Check it doesn't catch pages you want, such as tracked landing pages.
Why are tracking parameters only 'Minor'?
Google ignores most utm_ and click ID parameters and folds them into the clean URL, but other crawlers may not. Keep them out of internal links and use a canonical.
Does this crawl my site?
No. It analyses the list you paste; nothing is fetched or uploaded.
