Start a project

Tool 15 of 194 · AI visibility

robots.txt & llms.txt Change Watch

Check a site's robots.txt and llms.txt whenever you like and see exactly what changed since last time, and which AI crawlers it now blocks or allows.

Use the tool All developer tools Live · free
robots.txt & llms.txt Change Watch at a glanceIllustration: 1, robots.txt history; 2, processed in your browser; 3, Who got blocked.robots.txt & llms.txt Change WatchUser-agent: *Allow: /Sitemap: …# llms.txt1robots.txt history−++3Who got blocked2In yourbrowser
1 robots.txt history 2 processed on your device 3 Who got blocked

robots.txt & llms.txt Change Watch

Check a site

Each check reads https://HOST/robots.txt and /llms.txt once and saves a snapshot in this browser.

Check sends the host to leej.uk, which reads the two files as LeejTools/1.0 and keeps only a count. There's no background checking: snapshots stay in this browser, and the page reminds you after a week.

Space-separated. Verdicts are worked out for each path.

Or compare two versions

What changed

    Want the AI crawler view of your whole site? Run PingAI, which scores how well ChatGPT, Claude, Perplexity, Gemini and Copilot can reach and read it.

    robots.txt

    llms.txt
    
  
  
    Verdicts
    
  
    CrawlerCompanyPurposeBeforeNow
    

    

    Snapshot history
    
  
  
    Checkedrobots.txtSizellms.txtSize
    

    
    
    
    
    
  
    

  
    
    
    01
    What it does
    Fetches both files on demand, keeps snapshots in your browser, diffs them and re-tests 36 AI and search crawlers on old and new versions.
    • Both files fetched on demand, saved in your browser
    • Line-by-line diff against the last snapshot
    • 36 AI and search crawlers re-tested on old and new rules
    • Your own paths tested, not just the home page
    
    
    02
    Private by design
    Everything you paste or upload stays in your browser. Only when you press Fetch does leej.uk read the address you give, once, as LeejTools/1.0, keeping nothing but a daily count.
    
    
    03
    What you get
    A diff, verdict changes per crawler and a snapshot history, with a Copy and a Download button on every result.
    • robots.txt
    • llms.txt
    • AI crawlers
    • Monitoring
    
  
    

  
    
    
    Why it matters
    robots.txt changes without anyone deciding to change it. A CDN switches on a managed setting that blocks AI crawlers, a host adds a rule during maintenance, an SEO plugin update rewrites the file. Nobody notices until AI answers stop mentioning the site, or Googlebot gets a server error and stops crawling.
    

    This watch keeps a snapshot every time you check, compares it with the last one and re-runs every known AI and search crawler through the rules, old and new, for the paths you care about. It flags crawlers newly blocked or allowed, status changes and llms.txt edits. It doesn't run in the background or store anything on a server: snapshots live in this browser, and the page reminds you when a check is overdue.
    
    
    How it works
    1. InputTwo versions of robots.txt and llms.txt
    2. In your browserEvery known crawler run through both
    3. OutputChanged verdicts and a saved history
    Where it fits in a designer's workflow
    1. Plan
    2. Design
    3. Build
    4. Test
    5. Launch
    Catching a robots.txt change that blocks the wrong bots.
    By the numbers
    Crawlers blocked from the home page, before and after the sample editModel training: blocked before: 0 of 14; Opt-out token: blocked before: 0 of 2; Model training: blocked now: 3 of 14; Opt-out token: blocked now: 1 of 2Model training: blocked before0 of 14Opt-out token: blocked before0 of 2Model training: blocked now3 of 14Opt-out token: blocked now1 of 2
    Each of the 36 crawlers this tool knows was run through both sample robots.txt versions with the tool's own matcher. The edit blocks training crawlers but leaves AI search and fetch-for-a-person bots alone, so the site can still be cited. Source: OpenAI crawlers; RFC 9309 robots.txt.
    Show the numbers as a table
    CrawlerCompanyPurposeBefore (/, /guides/, /basket/)Now
    GPTBotOpenAItrainingallowed / allowed / blockedblocked / blocked / blocked
    ClaudeBotAnthropictrainingallowed / allowed / blockedblocked / blocked / blocked
    Google-ExtendedGoogletokenallowed / allowed / blockedblocked / blocked / blocked
    CCBotCommon Crawltrainingallowed / allowed / blockedblocked / blocked / blocked
    OAI-SearchBotOpenAIsearchallowed / allowed / blockedallowed / allowed / blocked
    ChatGPT-UserOpenAIuserallowed / allowed / blockedallowed / allowed / blocked
    Claude-SearchBotAnthropicsearchallowed / allowed / blockedallowed / allowed / blocked
    Claude-UserAnthropicuserallowed / allowed / blockedallowed / allowed / blocked
    anthropic-aiAnthropictrainingallowed / allowed / blockedallowed / allowed / blocked
    PerplexityBotPerplexitysearchallowed / allowed / blockedallowed / allowed / blocked
    Perplexity-UserPerplexityuserallowed / allowed / blockedallowed / allowed / blocked
    Google-CloudVertexBotGoogleuserallowed / allowed / blockedallowed / allowed / blocked
    GoogleOtherGoogletrainingallowed / allowed / blockedallowed / allowed / blocked
    GooglebotGoogleclassicallowed / allowed / blockedallowed / allowed / blocked
    Applebot-ExtendedAppletokenallowed / allowed / blockedallowed / allowed / blocked
    ApplebotApplesearchallowed / allowed / blockedallowed / allowed / blocked
    BingbotMicrosoftclassicallowed / allowed / blockedallowed / allowed / blocked
    Meta-ExternalAgentMetatrainingallowed / allowed / blockedallowed / allowed / blocked
    Meta-ExternalFetcherMetauserallowed / allowed / blockedallowed / allowed / blocked
    FacebookBotMetatrainingallowed / allowed / blockedallowed / allowed / blocked
    AmazonbotAmazonsearchallowed / allowed / blockedallowed / allowed / blocked
    BytespiderByteDancetrainingallowed / allowed / blockedallowed / allowed / blocked
    DuckAssistBotDuckDuckGouserallowed / allowed / blockedallowed / allowed / blocked
    MistralAI-UserMistraluserallowed / allowed / blockedallowed / allowed / blocked
    cohere-aiCoheretrainingallowed / allowed / blockedallowed / allowed / blocked
    YouBotYou.comsearchallowed / allowed / blockedallowed / allowed / blocked
    PetalBotHuaweisearchallowed / allowed / blockedallowed / allowed / blocked
    DiffbotDiffbottrainingallowed / allowed / blockedallowed / allowed / blocked
    TimpibotTimpitrainingallowed / allowed / blockedallowed / allowed / blocked
    ImagesiftBotImageSifttrainingallowed / allowed / blockedallowed / allowed / blocked
    OmgilibotWebz.iotrainingallowed / allowed / blockedallowed / allowed / blocked
    AI2BotAllen Institutetrainingallowed / allowed / blockedallowed / allowed / blocked
    YandexBotYandexclassicallowed / allowed / blockedallowed / allowed / blocked
    BaiduspiderBaiduclassicallowed / allowed / blockedallowed / allowed / blocked
    DuckDuckBotDuckDuckGoclassicallowed / allowed / blockedallowed / allowed / blocked
    LeejToolsleej.ukuserallowed / allowed / blockedallowed / allowed / blocked
    
    
    How to use it
    1. Enter the host and press Check now.
    2. Come back after changes or each week and check again.
    3. Read what changed: newly blocked crawlers are flagged first.
    4. Export the snapshots to keep a record.
    
    
    FAQ
    Why would robots.txt change on its own?
    CDNs (Cloudflare's managed robots.txt, for example), hosts and SEO plugins can write or edit it. This catches those changes.
    Does it check automatically?
    No. Each check is one you start, so there's no background polling and nothing stored on a server. The page reminds you after a week.
    What if robots.txt returns an error?
    A 404 means crawlers may crawl everything. A 5xx error makes Google treat the whole site as disallowed until it can read the file. Both are flagged.
    Where are snapshots kept?
    In this browser's IndexedDB. Clearing site data deletes them, so export them if you want a record.
    
    
    
  
    

    
  
    
    
    Work with me
    
    
    Want a site that gets these right?
    
    
    I build fast, accessible websites and web apps that pass these checks from day one. Tell me about your project and I'll reply personally with honest next steps.
    
    
    Start a project What I do
    
  
    

    

  
  
    All developer tools · Back to leej.uk