How to write llms.txt that AI tools can use

I keep both /llms.txt and a longer /llms-full.txt on leej.uk, and I built free tools for checking and writing them because I needed them on my own projects. What follows is that method.

What llms.txt is, and what it is not

The llms.txt proposal describes a markdown file at /llms.txt that starts with the project’s name, a short summary, and then sections of links, so a tool can fetch one predictable URL and get a trustworthy map of the site.

That is all it claims to be. It is not:

• a substitute for robots.txt. Crawlers still look at robots.txt to decide what they are allowed to request.
• a substitute for an XML sitemap. Sitemaps tell search engines which URLs exist and when they changed. Keeping a sitemap accurate is a separate job; the XML Sitemap Validator is a starting point.
• a ranking lever. There is no honest way to say that adding this file will move a page in Google, and I will not invent adoption numbers for any assistant.
• a licence to train on your content, or a way to opt out of training. Those are policy questions, handled with robots.txt, HTTP headers, or a usage policy. The AI Crawler robots.txt Tester is the starting point for that split.
• a dumping ground for marketing copy. If the file says you are the leading provider of something you cannot show, you have made the file less useful, not more.

It should be a truthful description of what is actually there, the same habit I describe in the privacy manifests guide, aimed at assistants rather than App Review.

What to put in the short index file

The short file is an index, not a book. Its job is to be easy to fetch, easy to parse, and easy to trust. A useful structure looks like this:

1. A heading with the site or project name. One h1. This is the name an assistant should use when it talks about you.
2. A short summary, often as a blockquote. Two or three sentences: who it is for, what you do, where you are based if that matters. No slogans.
3. A handful of official links. The website, a contact email, maybe a company number or a social profile you actually use. If an address or phone number is on the site, you can repeat it here; if it is not, do not invent one.
4. Sections of links, each with a one-line description. Group pages the way a helpful person would: products, documentation, policies, writing, tools. Every link should work. Every description should match the page.

Here is a complete, imaginary file for a small UK bookshop. Nothing in it is taken from a live site; it is here so you can see the shape:

# North Harbour Books

> An independent bookshop in Whitby, selling new and used books in the shop and by post across the UK. We do not sell e-books or subscriptions.

- Website: https://example.com/
- Email: desk@example.com
- Phone: +44 1947 000000

## Visit and order

- [Opening hours and finding us](https://example.com/visit): hours, parking and the street entrance.
- [Order a book](https://example.com/order): pay online and collect in the shop, or have it posted.
- [Book clubs](https://example.com/clubs): current lists and how to join.

## Policies

- [Privacy](https://example.com/privacy): what we keep when you order, and for how long.
- [Returns](https://example.com/returns): how to send a book back.
- [Accessibility](https://example.com/accessibility): step-free access and large-print stock.

A few choices in that file are deliberate, and they apply to a developer site as much as to a shop:

• Every claim is checkable. The summary says what they sell and what they do not. That second sentence is as useful as the first, because it stops an assistant inventing a service that is not there.
• Links are grouped by job, not by your CMS menu. Assistants do not care that “Visit” is a WordPress parent page.
• Descriptions are one line and specific. “Hours, parking and the street entrance” is better than “Learn more about visiting us.”
• Policies are present. Privacy, returns and accessibility are the pages a careful answer often needs, and they are the ones a homepage carousel usually hides.

On a site like this one, the same idea means listing the live projects, the developer tools, the articles and the legal pages, each with a sentence that matches the page, plus the contact details that already appear, including hello@leej.uk. If a section needs a paragraph of explanation, that paragraph belongs on the web page or in the fuller file described next. The index should stay a map.

When you also need llms-full.txt

The original proposal is one file. A second, longer file at /llms-full.txt is a pattern some sites, including this one, use when the short index is not enough for an assistant that wants the actual words on the pages.

The useful split is:

• /llms.txt — the map. Names, a summary, curated links, maybe a short note about how to contact you.
• /llms-full.txt — the readable text of the public pages, or a carefully chosen subset, in one place. This is closer to a clean export than to a brochure.

I generate the full file from the public HTML, not by pasting a second set of marketing lines. The moment the long file says something the live page does not, you have created a new source of truth, and it will drift. Rebuild it from the live pages, the same way I treat a release when I maintain an iOS app after launch.

A few practical limits help:

• Public, indexable pages only. Do not include /admin, thank-you pages, or anything you have marked noindex.
• The text a person can read, not the chrome. Drop navigation, scripts, cookie banners and form widgets. Keep headings, paragraphs, lists and captions.
• A cap per page is reasonable. A very long reference page does not need to be inlined in full if the short file already points at the live URL. Truncate cleanly, at a heading, and say that the rest is on the page.
• Do not include other people’s copyrighted work just because it appears in a comment or a quotation you would not want republished as your own text.

A ten-page brochure site with clear headings is often well served by the short index plus the pages themselves. Add the long file when you have many similar pages and you want an assistant to see the wording without crawling every URL. The llms-full.txt Builder turns clean page text into that file; the Markdown Mirror Generator is a useful way to see what a single page looks like once the markup is stripped.

Where the files should live, and how they are served

Put the short file at https://your-domain/llms.txt. That exact path is the one the proposal names, and it is the one tools will try first. If you also have a full file, put it at https://your-domain/llms-full.txt and mention it from the short file, so a client that only fetches the index still learns that the longer document exists.

A few serving details are easy to get wrong:

• Use your canonical host. If the site redirects www to the bare domain, or the other way around, the files should live on the same host, behind the same redirect.
• Serve them over HTTPS, with a Content-Type of text/plain or text/markdown. Do not wrap them in a HTML template or a download prompt.
• Let them be cached, but not for weeks. A short Cache-Control max-age, or an ETag, is enough. A stale llms.txt is worse than a slightly slower one; the Cache-Control Explainer covers the headers.
• Do not require JavaScript or a login. If a firewall or a challenge page sits in front of /llms.txt, the file might as well not exist for the crawlers you hoped would use it.

On a site served from a Cloudflare Worker, I treat these files as static assets in the same release as the HTML, so a deploy cannot ship a new article and an old index.

Allowing the crawlers that should read them

A file that is disallowed in robots.txt will not be fetched by a crawler that honours the file. That sounds obvious, and it is still the usual mistake. Decide who should be able to read the index — for many small UK sites, the search engines you already allow, plus the AI crawlers or fetchers you are comfortable with — then check the rules you already have.

A minimal, explicit approach is to allow the path for everyone you have not already blocked:

User-agent: *
Allow: /llms.txt
Allow: /llms-full.txt

Allow is only needed if a broader Disallow would otherwise hide those paths. If your robots.txt already allows the whole site, you do not have to list the files. If you have a rule such as Disallow: /, or you block unknown user agents, you do need the exception.

Two cautions:

• Do not open a path you meant to keep closed. If /llms-full.txt contains the text of pages that are not public, it should not exist, or it should be disallowed. The full file is not a back door.
• A rule for one user agent does not apply to another. Blocking a training crawler in robots.txt is a reasonable policy choice. Accidentally blocking Googlebot, or a fetcher that only retrieves a page when a person asked for it, is usually not. Test the real user-agent tokens rather than guessing. The AI Crawler robots.txt Tester and the robots.txt & llms.txt Generator are on this site for that check.

User-agent tokens change, so I will not pretend a list here will stay complete. Use a current directory of crawler names, and review the rules when you change policy.

Telling HTML pages that the file exists

The proposal does not require a special HTML tag, but a link in the document head is useful for humans, for your own tools, and for any client that looks for it. On leej.uk each page includes links of this shape:

<link rel="describedby" href="https://leej.uk/llms.txt" title="llms.txt">
<link rel="llms-txt" type="text/plain" href="https://leej.uk/llms.txt" title="LLM summary for AI bots">
<link rel="llms-full-txt" type="text/plain" href="https://leej.uk/llms-full.txt" title="Full site text for AI bots">

rel="describedby" is a long-standing way to point at a description of the site. The llms-txt and llms-full-txt relations are more specific and less widely standardised, so treat them as a hint, not a contract. They are still worth adding, because they are cheap and unambiguous.

Your robots.txt can include a comment, but comments are for humans. The short llms.txt should link to /llms-full.txt if you have one, and can also point at /robots.txt or /sitemap.xml. This site lists those under a “Site files” section. Including /llms.txt in the sitemap is optional and harmless if the file is public.

A worked example for a small UK site

Imagine a three-person studio in Manchester that builds and maintains a few client sites, and also publishes a small set of free tools and notes. The homepage explains the work. There is a projects list, a contact page, a privacy policy, and eight short articles. That is already enough for a useful llms.txt, and not so much that you need a database to maintain it.

A first version can be written by hand in an hour: list every indexable URL from the sitemap or the main navigation; drop anything thin, duplicated, or only meant for you; write one sentence per remaining URL that a stranger could use to decide whether to open it; and write the two-sentence summary last. If the summary needs a claim that is not on any page, either add it to a page or drop it from the file.

When the list grows past a handful, generate the full file from the HTML and keep the short file curated. That is the pattern I use here. The articles index at /articles is the human map; /llms.txt points at it and at the individual guides; /llms-full.txt carries the readable text so an assistant does not have to guess from titles alone. The same discipline I use when I write about small, focused Apple apps applies: do not list work that is not actually available, and say so when something is still in production.

Common mistakes

The usual problems:

• Writing a brochure instead of a map. Adjectives pile up, links stay vague, and the file cannot be used to answer a specific question.
• Letting the file go stale. A new tool or article ships, the index still describes last year’s site. Generate the full file, and add a reminder to update the short file as part of the same release. If you cannot keep it current, keep it small.
• Contradicting robots.txt or noindex. The short file must not advertise a URL you have asked crawlers not to fetch, or a page that is marked noindex and should stay out of search. Pick one story and tell it everywhere.
• Inventing facts to sound complete. Employee counts, “trusted by” lists, performance claims, and “best” language that would not survive a look at the site.
• Hiding the public contact details, or adding a personal email or phone number you do not want assistants to repeat.
• Serving HTML under the /llms.txt URL, or serving the file only after a JavaScript render. The file should be in the first response.
• Treating the file as an opt-out of training, or as an opt-in. It is neither. Use the proper policy mechanisms for that.

How to check the files

Before you announce anything, fetch the files the way a crawler would, not the way your browser shows a logged-in preview.

1. Request the exact URLs. curl -sSI https://your-domain/llms.txt should show 200, HTTPS, and a text content type. Follow redirects and confirm that both www and the bare host end on the same document.
2. Read the body. Is the markdown well formed? Do the links resolve? Do the descriptions still match the pages?
3. Check robots.txt for the user agents you care about, including a sample of AI crawlers, and confirm that /llms.txt is allowed for the ones you intend to allow.
4. Validate the file. The llms.txt Validator on this site checks the shape of the document and the obvious broken parts. It will not tell you whether a sentence is true; that is still your job.
5. Compare the full file to the live pages if you have one. A section that has no counterpart on the site is a bug.

Re-run the checks when you add a section of the site, in the same release as the thing it describes.

Key takeaways

• llms.txt is a short, accurate markdown index at the root of the site. It helps AI tools; it does not replace robots.txt, a sitemap or on-page SEO.
• Put only checkable facts and working links in the short file. Group them by the job a reader is trying to do.
• Add /llms-full.txt when you want the readable text of public pages in one place, and generate it from the live HTML so it cannot drift.
• Serve the files as text, on the canonical host, without a JavaScript wall, and make sure robots.txt allows the crawlers you want to read them.
• Link the files from the HTML head, and check them with a real HTTP request plus a validator whenever the site’s map changes.

If you want a second pair of eyes on a file like this, tell me about it.

Frequently asked questions

Written by Lee Johnson. Published 10 November 2026.