An llms.txt file is a plain Markdown document served at /llms.txt that gives AI systems a curated map of your site's most useful content — proposed by Jeremy Howard of Answer.AI in September 2024 as a workaround for HTML cluttered with navigation, ads and JavaScript that agents struggle to parse. Thousands of sites now publish one, but here's the part most guides skip: no major AI assistant has publicly confirmed it reads the file, and no search engine treats it as a ranking input. It's a proposal, not a standard. That gap between publisher enthusiasm and consumer adoption is exactly what you need to understand before deciding whether it earns a spot in your workflow.

What is an llms.txt file, and what is it actually for?

An llms.txt file is a Markdown file served at /llms.txt that gives AI agents a curated, link-based map of your site's most useful content — proposed by Jeremy Howard of Answer.AI in September 2024 at llmstxt.org.

The problem it targets is stated plainly in the spec: web pages are built for people, so HTML wraps the actual information in navigation, ads and JavaScript, and converting that back into clean text is "difficult and imprecise." Context windows, even in 2026, are still too small for most whole websites — every wasted token costs an agent time and money.

So instead of forcing a crawler to parse rendered pages, the file hands it a short, structured index: brief background on the site, guidance, and links out to detailed Markdown files. Because the format is precise Markdown rather than free-form prose, it can be processed with classical parsers and regex as well as read directly by an LLM. The spec also proposes clean .md versions of key pages alongside their HTML equivalents (e.g. page.html.md), discoverable via rel="alternate" type="text/markdown" link relations.

One thing, three names

If you've searched llms-txt, llms txt file or what is llms-txt and found seemingly different things — they're all the same proposal. The canonical repo is AnswerDotAI/llms-txt, Apache-2.0 licensed, created 1 September 2024. The hyphenated form is just the repo name; the file itself is always llms.txt.

The key caveat: llms.txt is a proposal, not a W3C or IETF standard, and no search engine has confirmed using it as a ranking or citation signal. That distinction shapes every practical decision about whether the file is worth your time — treat vendor claims that it "boosts AI Visibility Checker" accordingly.

Which AI assistants actually fetch llms.txt today?

None of the major AI assistant vendors — OpenAI, Anthropic, Google or Perplexity — has publicly documented fetching llms.txt to answer user queries. That is the honest answer, and it matters more than any llms txt generator tutorial: adoption is real on the publisher side and unverified on the consumer side.

What you can verify is publisher-side momentum, not assistant-side consumption. The v2 spec at llmstxt.org reports thousands of sites now publish an llms txt file, Mintlify generates one automatically for every docs site it hosts, and Chrome's Lighthouse now audits for llms.txt as part of its agentic browsing checks. That Lighthouse audit is the closest thing to browser-vendor endorsement the format has.

The strongest signal: the labs publish it themselves

The AI labs maintain llms-txt files for their own developer documentation — OpenAI, Anthropic and Gemini all serve one. Publishing is not the same as consuming, but it tells you the teams closest to agentic tooling consider the format worth maintaining for their own properties.

The distinction that resolves this apparent contradiction is which AI is fetching. Chat assistants answering consumer queries have not documented llms.txt support; coding agents and custom tools fetching documentation on demand are the use case the v2 spec was rewritten around, and it is where the format demonstrably fits.

What this means for your decision

• Treat an llms txt file as low-cost infrastructure for agentic and developer-tool traffic, not a ranking or citation lever for ChatGPT or AI Overviews.
• Don't expect measurable assistant-citation lift from publishing one — no vendor has committed to that, and practitioner threads asking for measurable impact reflect exactly this uncertainty.
• If your site is documentation-heavy, the labs' own behaviour is your precedent; if it's a marketing site, the case is far weaker.

llms.txt vs robots.txt vs sitemap.xml: what does each one actually do?

Each file speaks to a different layer: robots.txt controls what crawlers may fetch, sitemap.xml lists every URL for discovery, and llms.txt curates the pages worth an LLM's attention.

They are complementary, not competing. Confusing them is the most common mistake we see when teams first ship an llms txt file — usually treating it as a blocking mechanism, which it is not.

robots.txt — the permission layer

robots.txt tells crawlers what they may not fetch. It is directive-based, decades old, and honoured by the major AI crawlers that declare a user agent (GPTBot, ClaudeBot, Google-Extended). It says nothing about which pages are good — only which are off-limits.

sitemap.xml — the discovery layer

sitemap.xml is a machine-readable inventory of every indexable URL on your site. It is comprehensive by design and carries no editorial judgement: your best pillar page and your thinnest tag archive sit in the same list, unranked.

llms.txt — the curation layer

An llms.txt file is a human-written Markdown shortlist that says "these are the pages that matter, and here is why," per the llms.txt spec. The spec also proposes clean .md versions of key pages and an expanded llms full txt variant, so an agent can pull concise, expert-level content without parsing navigation, ads and JavaScript. Curation is the entire point — a dumped sitemap in Markdown clothing defeats it.

File	Layer	Format	Answers the question
robots.txt	Permission	Directives	"What may you not fetch?"
sitemap.xml	Discovery	XML	"What exists here?"
llms.txt	Curation	Markdown	"What matters here, and why?"

llms.txt replaces neither of the other two. If you have to pick one to fix first, fix robots.txt (it is enforced) and your sitemap (it is actively consumed) before writing llms.txt (it is optional and, today, inconsistently used by AI systems).

What goes in a good llms.txt file? (format and example)

A good llms.txt file is a short Markdown document: an H1 with the site or project name, an optional blockquote summary, then H2 sections of curated link lists with one-line descriptions. That structure is deliberate — it is precise enough for parsers and regex to process programmatically, per the llms.txt spec, while staying readable for humans and models alike.

The canonical structure

The spec defines four ordered parts, only the first of which is mandatory:

• H1 — the site or project name (the only required element)
• Blockquote — a one-paragraph summary of what the site is and who it serves
• Optional prose or lists — brief context or guidance an agent should know before following links
• H2 sections — Markdown link lists, each link followed by a one-line description of what's behind it

A minimal llms.txt example looks like this:

# IT Master

> Autonomous AI content platform: GEO, E-E-A-T and technical SEO guides for teams scaling expert content.

## Docs

- GEO guide: How AI engines select and cite sources
- Content validation: Cross-model adversarial checking explained

Every link should carry that one-line description — a bare URL forces the model to fetch the page just to learn whether it's relevant, which defeats the point of a curated index.

The .md companion convention

V2 of the spec pairs the index with clean Markdown versions of your key pages, served at the same URL with .md appended (page.html.md) or the extension swapped (page.md); directory URLs use index.md. This gives agents the content without navigation chrome, ads or JavaScript to strip.

You then expose those files with two standard link relations: rel="alternate" type="text/markdown" pointing to the Markdown version, and rel="describedby" pointing back to the covering llms.txt. Both can ship as HTML <link> elements or an HTTP Link: header — the header form also works for non-HTML resources like the .md files themselves.

Placement and scope in v2

An llms.txt file no longer has to sit at the site root: it can live at any path and covers everything beneath it, so /docs/llms.txt describes all of /docs/. For large sites, that means separate, tightly-scoped files per product area rather than one sprawling root index.

llms-full.txt: the full-content variant

llms-full.txt — the "llms full txt" people search for — inlines the complete content of your key pages into a single file, rather than linking out to them. The trade-off is directness versus size: an agent gets everything in one fetch, but a large llms-full.txt can outgrow the practical context budget the standard index format was designed to respect. Documentation platforms such as Mintlify generate both variants automatically, which is the low-effort route if your docs already live on one.

How do you generate, validate and check an llms.txt file?

You can hand-write an llms.txt file in about fifteen minutes for a small site, or generate one with tools like Firecrawl or Mintlify. Documentation platforms increasingly emit one automatically — Mintlify generates the file for every docs site it hosts, per the v2 spec, and OpenAI, Anthropic and Google all publish llms.txt files for their own developer docs.

Why hand-writing beats a generator for editorial sites

A generator crawls your sitemap and lists what it finds. But the value of an llms txt file is the selection — which twenty pages actually matter — and the one-line description after each link. That editorial judgement is exactly the part no llms txt generator can infer, because it lives in your head, not your HTML.

For a docs platform with a clean information architecture, automated generation makes sense. For an editorial site at itmaster.uk scale, hand-curating the file forces the useful exercise: deciding what you'd want an assistant to read first.

What a checker can — and cannot — verify

An llms-txt validator can genuinely confirm four things:

• The file resolves at the correct path (/llms.txt at the root, or a sub-path per the v2 spec)
• It serves the right content type as plain Markdown
• The Markdown structure is valid per the format reference — H1 title, blockquote summary, H2 link sections
• Every listed link resolves rather than 404s

What no llms txt checker can confirm is whether any assistant actually reads it. Validation proves the file is well-formed; it cannot prove consumption. Chrome's Lighthouse now audits for the file in its agentic browsing checks, which is a formatting signal, not a usage guarantee.

How to "submit" an llms.txt file

You don't. There is no submission endpoint — no Search Console equivalent, no ping URL, no registry. You publish the file at your root and crawlers either fetch it or they don't. Anyone selling llms.txt submission is selling nothing.

Is llms.txt Worth It in 2026?

An llms.txt file is worth adding if your content is documentation-heavy or citation-dependent, but it will not directly move rankings — no major AI vendor has confirmed using it as a ranking or retrieval signal. Treat it as a low-cost bet, not a growth lever.

The honest picture in 2026 is mixed. On one side, adoption is real: thousands of sites publish an llms-txt file, Mintlify generates one automatically for its documentation customers, and Chrome's Lighthouse now audits sites for llms.txt as part of its agentic browsing checks. The AI labs themselves ship one for their own developer docs — OpenAI, Anthropic and Gemini all publish live files.

On the other side, practitioners openly debate whether it changes anything measurable. Forum threads with titles like "Is anyone actually seeing measurable impact from adding llms.txt?" capture the skepticism, and it is a fair question — the spec is a proposal (now on v2 at llmstxt.org), not a standard any crawler is obliged to honour.

Our position at IT Master: publish one, spend under an hour on it, and put your real GEO effort where citation actually happens — extractable answer-first sections, first-party data and cross-model validated accuracy. The mechanism that gets you cited in ChatGPT or Perplexity is clean, quotable, verifiable content; an llms txt file just makes that content cheaper for an agent to find and parse.

Where the payoff is most likely

• Developer docs and APIs — coding agents fetch docs constantly; this is exactly the use case the spec was written for.
• Deep sites with heavy templates — a curated markdown index saves agents from parsing navigation, ads and JavaScript.
• Thin brochure sites — lowest payoff; your effort is better spent on the content itself.