llms.txt generator — the file and its robots.txt block
Fill it in and take both outputs, shaped so they do not carry the most common robots.txt defect.
Nothing leaves your browser. What you type here goes nowhere: no server call, no analytics, no storage. The files are assembled on your own machine and reach your disk only when you click download.
llms.txt is a Markdown file at the root of a domain listing the pages that matter and what each covers in one sentence. This form produces it in the correct structure: a title, a one-sentence summary, then links with descriptions. Alongside it, the tool generates a robots.txt block for the named AI crawlers — because the two files do different jobs, and most confusion comes from conflating them: llms.txt informs, robots.txt governs.
One per line: URL — description. The description is the point of the file; do not omit it.
Write full paths rather than patterns. The generator strips asterisks — in robots.txt they span the slash.
Enter a company name and at least one URL, and the files appear here.
What to watch in the output
Do not leave the description empty and do not paraphrase the title: a bare link list adds nothing a sitemap does not, and the file's value lives in the descriptions. For exclusions, write full paths rather than patterns — the generator deliberately strips any asterisk you type. There is a reason: in robots.txt an asterisk SPANS the slash, so a rule shaped like "/*/private" also catches "/en/products/private". That defect hit us in production and walled four live product pages off from every AI crawler.
Frequently asked questions
- Do you store what I type?
- No. There is no server call and no analytics on this page: the files are assembled in your browser and reach your disk only if you download them. This is verifiable from the page source.
- Why generate a robots.txt block too?
- Because the two files get conflated constantly. llms.txt informs — it says what matters; robots.txt governs — it says what may be fetched. If you want to exclude something from collection, only robots.txt can do that; llms.txt cannot.
- Why do you strip asterisks from exclusions?
- Because in robots.txt an asterisk spans the slash, so a pattern rule catches far more than intended. This happened to us in production: four live product pages dropped out of reach of every AI crawler because of one such pattern. Write full paths instead.
- Is this file enough on its own?
- No. The usefulness of llms.txt is currently unproven, and it certainly does not replace a sitemap or proper crawlability. If you have to choose, fix the things collectors are documented as using first.