glossary · llms.txt
llms.txt: what the file is, and what it cannot do
A proposed convention: a plain markdown file at your site root listing the pages you want a model to read. What goes in it, what it can reasonably do, and the claims to ignore.
llms.txt is a proposed convention: a markdown file at a site's root that lists its most important pages, each with a short description, so an AI model can find and read them without crawling the whole site.
reviewed 2026-09-02 · by the IT Master editorial team · how we check facts
What llms.txt looks like in practice
The file sits at /llms.txt, the root of a domain, and it is ordinary markdown rather than a new syntax. The shape from the original proposal is short:
- An H1 carrying the site or product name.
- A blockquote of one or two sentences saying what the site is.
- Optional paragraphs of context: scope, units, anything a reader needs before the links.
- H2 sections, each holding a bullet list of links, with one line per link on what that page covers.
- A closing section named Optional, which a model may skip when its context budget is tight.
Two companions travel with it. An llms-full.txt holds the whole corpus concatenated into a single markdown file, which suits documentation and little else. And clean markdown versions of individual pages, usually the same URL with .md appended, which several documentation platforms now serve.
Serve it as text/plain or text/markdown, use absolute URLs so a model reading the file on its own can still fetch what it lists, and keep it curated rather than exhaustive. A file listing every URL you own is a sitemap with worse tooling.
An llms.txt is an orientation file, not a permission file: it suggests where a model should start reading, while robots.txt decides whether that model may read anything at all.
Why it matters in 2026
The case for the file is cost, not evidence. It is cheap to write, it breaks nothing, and there are two situations where it can earn its keep: an agent pointed at your domain that has to decide where to start, and a tool that wants your content as markdown rather than as HTML wrapped in navigation.
The case against overselling it is simpler. No major search or assistant vendor documents llms.txt as an input to retrieval or ranking. Until one does, treat any claim that the file lifts rankings as a guess, and treat your own access log as the only evidence you have about who fetches it. That may change. It has not changed yet.
The useful part is the exercise. Choosing which pages belong in the file forces a decision about which are canonical, which are duplicates, and which you have quietly stopped maintaining. That decision helps the things that are proven: crawlable HTML, one page per question, an answer near the top. Treat the file as a tidy side effect of LLM SEO, not as the work itself.
How we generate one, and what we claim for it
The engine renders an llms.txt for every site it publishes to, from the same rows that produce that site's sitemap.xml and RSS feed: the articles that actually went live. One source of truth means the three indexes cannot drift apart, and a malformed row is skipped rather than allowed to corrupt the file. The output is the conventional shape: the site name as a heading, a one-line summary, then each published article as a titled link to its canonical URL.
Serving it is the site's job. A Next.js site mounts one route handler from the SDK, described on the Next.js integration page. Pull-feed and webhook consumers fetch the same text from the API and serve it at their own root.
We do not claim the file earns citations. Emitting it costs nothing, because the engine already knows every canonical URL it published, and the measurement that matters runs elsewhere: the nightly loop watches Search Console and whether assistants such as GPT and Perplexity cite the site at all.
The free AI Crawler Check reports whether a domain serves one, alongside its robots.txt status and whether a sitemap is declared.
Common misunderstandings
That it grants or blocks access. Permission is robots.txt. llms.txt says nothing about who may fetch what, so if GPTBot is disallowed, a well-written index changes nothing.
That it replaces a sitemap. A sitemap is exhaustive, machine-generated and meant for crawlers. This file is curated, short, and written for a reader with a limited context window. You still need the sitemap.
That publishing it means something reads it. Grep your access log for /llms.txt and look at the user agents before you assume anything. Checkers and SEO tools fetch that path too, and they are easy to mistake for interest.
That bigger is better. A generated dump of every URL you own is worse than a short curated list, because the point is to spend a model's attention on the pages you keep current.
That it rescues a page nobody can read. If the main content only appears once JavaScript runs, or the answer is buried in paragraph nine, the index points at a page that still cannot be quoted.
An llms.txt cannot grant access, replace a sitemap, or make an unreadable page quotable; it can only tell a model which of your readable pages to start with.
questions people ask
Does Google use llms.txt?
No. Google does not document llms.txt anywhere in its crawling and indexing documentation, and no other major search or assistant vendor lists it as a retrieval or ranking input. That is the position as of this page's review date, and it is worth re-checking, because the convention is young and adoption is uneven. Publishing one is still defensible on cost: it is quick to write, it breaks nothing, and an agent pointed straight at your domain has to start somewhere. What it will not do is move rankings. Anyone selling it as an AI ranking factor is guessing.
What should go in an llms.txt file?
A heading with your site name, one or two sentences saying what the site is, then grouped lists of links with a short note on each. Include the pages you would hand a new colleague on their first day, not everything you have ever published. Use absolute URLs so a model reading the file on its own can still fetch them. Put anything skippable in a final section named Optional. Then keep it current, because a file full of links to pages you no longer maintain is worse than no file at all.
Is llms.txt the same as robots.txt?
No, and confusing the two wastes effort. robots.txt is an access convention: it names crawlers and tells them which paths they may fetch, and the well-behaved ones obey it. llms.txt is an orientation convention: it assumes a reader is already allowed in and suggests where to start. Neither is enforced by anything but goodwill, but only robots.txt is widely honoured today. If your goal is to keep AI crawlers out, edit robots.txt. If your goal is to help one that is already welcome, write the index.
How do I tell whether anything actually reads mine?
Filter your server access log for requests to /llms.txt and group them by user agent and date. Expect automated checkers and SEO tools in that list, so look specifically for named AI crawlers and for the assistant user agents that fetch pages on a user's behalf. Then compare it against the outcome you care about, which is whether assistants name your site when someone asks a question in your subject area. Fetches are the input; being cited is the result, and the two do not move together reliably.
See what search engines and AI assistants find on your site
Free, no account. Type your address and we show you what is missing and what we would write first.