In September 2024, Jeremy Howard, the co-founder of Answer.AI and a well-known figure in machine learning, proposed a small idea with a big ambition: a standard file, called llms.txt, that websites could publish to help AI systems understand their content. The proposal spread quickly through the AI and SEO communities, and within months people were arguing about whether it was the next robots.txt or a solution looking for a problem. That argument is still live, which is exactly why it is worth understanding what the file is before you decide where you stand.
What is llms.txt? It is a proposed standard file, placed at the root of your website, that provides a clean, curated, markdown-formatted guide to your site’s most important content for AI systems and language models. The idea is to give AI crawlers a simple, readable map of what matters on your site, rather than making them parse messy HTML full of navigation, ads, and scripts. Think of it as a curated table of contents written for machines. Whether llms.txt becomes a lasting standard or fades is not settled, but understanding what llms.txt is, how it differs from files you already know, and whether it is worth adding, is a reasonable use of an afternoon. This is the guide.
What llms.txt actually is
An llms.txt file is a markdown document at yoursite.com/llms.txt that lists and links your most important content in a form a language model can read easily. It typically opens with your site’s name and a short description of what you do, then organizes links to key pages under clear headings, sometimes with a sentence explaining each. The format is deliberately simple, because the whole point is to be easy for a machine to parse.
The reasoning behind it is that a normal web page is a poor thing to feed a language model. It is wrapped in navigation, sidebars, cookie banners, scripts, and markup that have nothing to do with the actual content, and a model has to work to find the substance. An llms.txt file strips all of that away and hands the model a clean, curated view of what you consider important. When people ask what llms.txt is at a practical level, this is the answer: it is you telling AI systems, in a format they find easy to read, here is what my site is about and here is the content that matters most. Whether they listen is a separate question, which the next sections take up.
How llms.txt differs from robots.txt and sitemaps
The natural comparison is to files you already run, and the differences are the point. Robots.txt tells crawlers what they are allowed to access; it is about permission and exclusion. A sitemap lists your URLs so search engines can discover them; it is about completeness and discovery. Neither one curates or explains. They do not tell a machine what matters or what your content means. They just gate access and enumerate pages.
llms.txt is a different job: curation and comprehension. It is not trying to list every URL or block any crawler. It is trying to give an AI system a curated, explained guide to your best content, so the machine understands your site rather than merely finding its pages. That is why it exists alongside the older files rather than replacing them. You keep robots.txt for permission and a sitemap for discovery, and llms.txt adds a layer neither one provides: a human-curated, machine-readable summary of what your site is and where its value lives. Understanding what llms.txt is means seeing it as complementary, a comprehension aid rather than a crawl-control or discovery tool.
The crawl-to-cite path
To judge whether llms.txt is worth your time, it helps to see where it sits in the larger journey from crawl to citation, which I call the crawl-to-cite path. For an AI engine to cite you, four things have to happen in order: the engine has to find your content, read it, understand what it means, and then trust it enough to use it. Discovery, reading, comprehension, trust. Each link is a place you can help or hinder the engine.
llms.txt targets the comprehension link, and only that link. A sitemap helps discovery. Clean, crawlable pages help reading. Clear answers and corroboration help trust. llms.txt sits in the middle, helping the engine understand what your site is about and which content is important, once it has found and read you. That framing is useful because it sets honest expectations. llms.txt will not fix a site the engines cannot crawl, and it will not manufacture the trust that comes from corroboration. What it can do is make the comprehension step easier, so an engine that has found and read your site grasps your content and priorities faster. It is one link in the crawl-to-cite path, valuable in its place, and no substitute for the others.
Should you add an llms.txt file?
Here is the honest cost-benefit. The cost is low. Writing an llms.txt file is a short exercise: your site name, a description, and curated links to your important pages under clear headings. If you understand your own content, you can produce a solid one in an hour, and maintaining it is light. The benefit is real but uncertain, because llms.txt is a proposed standard, not a universally enforced one, so some AI tools use it and others ignore it entirely today.
That balance points toward adding it, with clear eyes. The downside is an hour of work and a file that some crawlers may not read. The upside is that you make your content easier to understand for the tools that do read it, and you are positioned if adoption grows. It is a low-cost, forward-looking move, not a magic visibility switch. The mistake to avoid is treating llms.txt as the thing that will get you cited in AI answers. It is a small comprehension aid, and it sits downstream of the discovery, reading, and trust work that actually determines whether an engine names you. Add it because it is cheap and sensible, not because it will carry your AI visibility on its own.
What a good llms.txt file looks like in practice
To make it concrete, picture how a well-run business software site might structure its file. It opens with the company name and a one-line description of what the software does and who it serves. Then it groups links under headings a machine can scan: documentation, key product pages, pricing, guides, and support resources, each link followed by a short sentence explaining what the reader will find. Nothing fancy, nothing padded, just a clean, curated map of the content that matters, written so a language model can absorb it at a glance.
That example captures the spirit of llms.txt: curation over completeness, clarity over cleverness. You are not dumping every URL; you are choosing the content you most want AI systems to understand and presenting it cleanly. Build yours that way, keep it current as your important content changes, and treat it as one modest, sensible layer in a larger AI visibility effort. If llms.txt becomes the standard its proposers hope for, you will be glad you had one early. If it does not, you spent an hour making your best content easier for machines to understand, which was never a bad idea regardless of which file carried it.