llms.txt Explained: What It Is, What It Isn’t, and How to Write One

Someone on your team has probably forwarded you a link about llms.txt with a note saying we should add this. Before you put it in a sprint, it’s worth understanding what the file does, what it demonstrably doesn’t do, and why it is still probably worth twenty minutes of your time.

The short version: llms.txt is a proposed convention, not a standard, and no major AI system has publicly committed to reading it. It costs almost nothing to publish and may help; it won’t make you visible in ChatGPT on its own. Both halves of that sentence matter.

What is llms.txt?

llms.txt is a plain Markdown file published at the root of your domain, at https://example.com/llms.txt, that gives AI systems a curated, machine-friendly map of your most important content. It was proposed by Jeremy Howard in 2024 as an analogue to robots.txt and sitemap.xml, aimed at language models rather than crawlers.

The reasoning behind it’s sound. Web pages are cluttered with navigation, cookie banners, scripts and interface chrome, and context windows are finite. A concise index that says here is what this organisation does and here are the canonical pages for each topic is genuinely useful to a system trying to understand a site quickly.

What llms.txt is not

This is where most of the confusion lives, so be direct about it.

• It’s not an official standard. There’s no W3C or IETF specification, only a community proposal that has gained mindshare.
• It isn’t robots.txt. It grants no permissions and blocks nothing. Crawler access is still controlled entirely by robots.txt and your server rules.
• It’s not confirmed to be read by the major engines. Google has said it doesn’t use it. Others have not committed publicly either way.
• It is not a ranking or citation mechanism. Nothing in the file causes you to be quoted.
• It isn’t a substitute for well-structured pages. If the pages it points to are unextractable, the index has nothing worth indexing.

Treat it the way you would treat an emerging convention with low cost and uncertain payoff: publish it, keep it accurate, and do not build a strategy on it.

What does the format actually look like?

The convention is deliberately minimal. An H1 with your organisation name, an optional blockquote summary, optional prose context, then H2 sections containing Markdown link lists with short descriptions. An optional section is a conventional signal that its contents can be skipped when context is tight.

# Northfield Analytics

> Independent data and reporting consultancy working with mid-market
> manufacturers, healthcare providers and public sector bodies in the UK.

Northfield Analytics designs reporting systems, migrates legacy data
warehouses, and trains internal teams to maintain them. Founded 2014,
offices in Leeds and Bristol.

## Services

- [Data warehouse migration](https://example.com/services/migration.md): Phased migration approach, typical timelines and risk controls.
- [Reporting design](https://example.com/services/reporting.md): How we scope, build and hand over operational reporting.
- [Team training](https://example.com/services/training.md): Curriculum and delivery formats for internal analytics teams.

## Guides

- [Choosing a warehouse platform](https://example.com/guides/platforms.md): Selection criteria and trade-offs, vendor-neutral.
- [Migration readiness checklist](https://example.com/guides/readiness.md): What to verify before committing to a migration.

## About

- [About Northfield](https://example.com/about.md): History, leadership, sectors served.
- [Contact](https://example.com/contact.md): Offices, enquiry routes, response times.

## Optional

- [Press coverage](https://example.com/press.md): Third-party articles and mentions.

Two details are easy to get wrong. Use absolute URLs, since a model reading the file in isolation cannot resolve relative paths. And write descriptions as real information rather than marketing copy, because the description is often the only thing a system reads before deciding whether to fetch the page.

How do you write a good one?

1. Be selective. Fifteen to forty links, not your whole sitemap. Curation is the entire value.
2. Lead with identity. The summary should let a reader answer who this is, what they do, and for whom without following a single link.
3. Group by user intent. Services, guides, about, policies. Mirror how someone would ask, not how your CMS is organised.
4. Describe each link concretely. One sentence stating what the page contains.
5. Keep it current. A file pointing at retired services is worse than none, because it actively misinforms.
6. Serve it as plain text. Content-Type: text/plain; charset=utf-8, at the domain root, no authentication.

Some sites also publish Markdown versions of individual pages at a .md suffix, as in the example above, and a larger llms-full.txt containing complete content rather than links. Both are optional and only worth doing if you can maintain them.

So should you publish one?

Yes, with the right expectations. It is a small, low-risk file that forces a genuinely useful exercise: deciding which twenty pages actually represent your organisation and describing each one plainly. Many teams find the drafting more valuable than the artefact.

What it should never do is displace the work that demonstrably matters. Confirm AI crawlers aren’t blocked, serve content in HTML rather than only after JavaScript, structure pages so passages can be extracted, publish consistent entity facts, and implement correct schema. Those change citation outcomes; llms.txt might.

A reasonable sequencing rule: publish llms.txt in an afternoon, then spend the following month on page structure. Score the pages you link to with the AI citability scorer, and track whether anything changes with the ARIA citation tracker.

If you would like the whole layer assessed and implemented, see our AI visibility practice or start a project.

Frequently asked questions

Does Google read llms.txt?

Google has stated it doesn’t use the file. Other AI providers haven’t made public commitments either way, so treat any claim of confirmed support with scepticism.

Can llms.txt stop AI systems using my content?

No. It carries no permissions and blocks nothing. Access control belongs in robots.txt, your WAF rules, and your terms of use.

Where does the file go and what should it be named?

At the root of the domain, exactly /llms.txt, served as plain text without authentication. Subdirectory or renamed variants will not be found by anything looking for the convention.

Is llms.txt worth doing if nothing confirmed reads it?

For most sites yes, because the cost is an afternoon and the curation exercise has independent value. Just don’t let it substitute for structural and technical work that measurably affects citation.