Want the AI crawler view of your whole site? Run PingAI, which scores how well ChatGPT, Claude, Perplexity, Gemini and Copilot can reach and read it.
01
What it does
A searchable reference of AI crawlers, fetchers and agents: robots token, purpose, IP list, documentation and opt-out method.
Rendered from scripts/ai_bots.py (single source of truth for the robots tester, log analyser, firewall builder)
Fields: token, UA pattern, owner, category, honours robots (documented yes/no/unknown), IP list URL, docs URL, last checked date
Filters, CSV/JSON export, 'copy robots block'
Each entry gets an anchor for deep links
02
Private by design
Everything happens in your browser. Nothing you type, paste or upload is sent anywhere, and it keeps working offline once the page has loaded.
03
What you get
A filtered table, plus CSV and JSON, with a Copy and a Download button on every result.
AI crawlers
User agents
Directory
Why it matters
A searchable reference of AI crawlers, fetchers and agents: robots token, purpose, IP list, documentation and opt-out method.
Rendered from scripts/ai_bots.py (single source of truth for the robots tester, log analyser, firewall builder). Fields: token, UA pattern, owner, category, honours robots (documented yes/no/unknown), IP list URL, docs URL, last checked date. Everything runs in your browser; nothing you paste is sent unless you press Fetch.
How to use it
Paste or type the input the tool asks for.
Read the summary and the table or preview.
Adjust the options if you need a different result.
Copy or download the output. Nothing leaves this page.
FAQ
Is my input uploaded?
No. The work runs in your browser and nothing you type, paste or upload is sent anywhere.
Will this change my site for me?
No. It shows results and, where it writes rules or files, you copy them and apply them yourself.
How does this relate to PingAI?
Each bot row links to PingAI to test it against your site.
I build fast, accessible websites and web apps that pass these checks from day one. Tell me about your project and I'll reply personally with honest next steps.
