Skip to content
AVE StudioFor marketing agenciesENHU

Cookbook

llms.txt: format, example, and whether it is worth your time

A cheap, well-specified file that nobody can currently prove anyone reads — and that is exactly what you should know before writing one.

Updated ·

llms.txt is a proposal: a Markdown file at the root of a domain listing the pages that matter, each with one sentence saying what it covers. The idea borrows its shape from sitemap.xml and robots.txt, with one difference that matters — it is a community proposal rather than a standard, and its job is orientation rather than access control. The honest status in August 2026: the format is well defined and cheap to produce, and no major AI vendor's public documentation describes consuming it. That does not make it pointless. It makes its usefulness unproven, and anyone guaranteeing you otherwise is claiming more than they can know. This recipe covers the format and what belongs in the file, then tells you when the half hour is worth spending at all.

The procedure

  1. Decide what you are actually pointing at

    llms.txt is not a copy of your sitemap. A sitemap aims for completeness; llms.txt aims for selection — what matters, and why. List the ten to thirty pages you would hand to a person trying to understand you: category pages, methodology, glossary, evidence. If a page does not fit the sentence "I am showing this because…", it does not belong here.

  2. Follow the format

    Markdown, at the root, named `llms.txt`. One `#` heading with the site name; a `>` blockquote with a one-sentence summary; then `##` sections containing links in the form `- [Title](URL): one sentence of description`. The description is the point rather than decoration — a bare list of links tells a reader nothing a sitemap does not. If a section grows past fifteen or twenty entries, split it; readability is the goal here, not coverage.

  3. Generate it — never hand-maintain it

    The most important step, and the one most often skipped. A hand-written llms.txt is correct exactly as long as someone remembers it exists; the first new page makes it stale, and from then on it is worse than nothing, because it states a confidently incomplete picture. Derive it from the same list your sitemap is built from and produce it during the build. On this site one path array feeds both, so they cannot drift apart — that is not elegance, it is the only way to stop relying on memory.

  4. Write descriptions, not restated titles

    Do not paraphrase the title in the description field. Say what the page answers and for whom. A good description stands alone: reading it tells you whether to open the page. Avoid marketing adjectives — "industry-leading solution" carries no information, while "how the measurement works: what we measure, how many runs, and what we do not promise" places the page precisely.

  5. Publish it, then verify it is really being served

    The file should answer at `/llms.txt` with a 200 and a `text/plain` or `text/markdown` content type. Verify by request rather than from the repository: a file's presence in version control does not prove the server serves it — a failure class we hit in production on a different static file. If a CDN or edge layer sits in front, check through that too.

  6. Wire it into the process so it stays current

    If generation is part of the build, this costs nothing. If it is not, add a check that fails as soon as the committed file differs from what the generator would produce today. The goal is not to be right once; it is to make silent staleness impossible — the same principle that applies to every other generated artifact.

Frequently asked questions

Is llms.txt required?
No, and it is not a standard. It is a community proposal that anyone may follow; there is no standards body behind it and no vendor requiring it. Anyone telling you it is required is presenting a proposal as a specification.
What is the difference between llms.txt and robots.txt?
They do different jobs. robots.txt governs access: what may be fetched. llms.txt informs: what matters and what it covers. Major collectors document honouring robots.txt; none document reading llms.txt. Neither substitutes for the other, and they are not in competition.
Can it replace my sitemap?
No, and it is not meant to. A sitemap is built for machine discovery, aims at completeness, and is documented as used by search engines. llms.txt is a curated selection with context, for human and machine readers alike. Keep both; if you can only maintain one, keep the sitemap, because its effect is demonstrable.
What goes in it if the site is small?
The same rule applies: whatever you would hand to someone trying to understand you. Five good entries beat thirty you could not justify. If the whole site is ten pages, list ten — the value is in the descriptions rather than in the filtering.
Can a bad llms.txt hurt?
Not directly, but a stale one states a confidently wrong picture — pointing at pages that no longer exist, for instance. That is what step three is about: the hand-maintained version is not bad because it is hard, it is bad because it decays silently.

Sources

  1. The llms.txt proposal. https://llmstxt.org/
  2. OpenAI — Bots (the documentation does not mention consuming llms.txt). https://developers.openai.com/api/docs/bots
  3. Anthropic — crawler documentation (likewise silent on it). https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler