Why it matters
AI companies use different crawlers for different jobs. Some collect pages to train models; others find pages to show and link in AI search answers, which can send you visitors. Blocking everything with one rule can take you out of ChatGPT and Perplexity answers as well as their training data.
This generator keeps the two apart, so you can, for example, allow AI search and block training. It also writes an llms.txt: a short, plain Markdown summary of your site that AI assistants can read quickly.
How to use it
- Enter your site address, the paths to keep private and your sitemap.
- Pick a preset for AI crawlers, or set each one.
- Fill in the llms.txt summary and links.
- Download both files and put them in your site's root folder, then check the result with PingAI.
FAQ
Does blocking Google-Extended affect my Google ranking?
No. Google-Extended only controls whether Google uses your pages for Gemini. Google Search uses Googlebot, which this setting doesn't touch.
Will every AI crawler obey robots.txt?
The big companies say their crawlers do. Fetchers that open a page because someone asked (such as ChatGPT-User) may not, and some smaller crawlers ignore it. Use a firewall rule if you need a hard block.
Is llms.txt an official standard?
Not yet. It's a proposal from llmstxt.org that a growing number of sites and tools use. It costs nothing to add and doesn't affect search engines.
Where do the files go?
At the root of your domain: example.co.uk/robots.txt and example.co.uk/llms.txt.
