An llms.txt file is a short, plain-text guide you place at the root of your site, written in markdown, that points an AI reader to your most important pages and explains what they are. That is the whole idea. When a crawler or an answer engine wants to understand your site, it usually has to infer your priorities from a sprawling sitemap or by guessing from your navigation. An llms.txt file removes the guessing by handing over a curated map: here are our key pages, here is what each one covers, start here. Creating one takes an afternoon, and it is one of the cheapest things you can do to make your site legible to the models that increasingly read it.
The reason to bother is asymmetry. The cost of adding the file is near zero, a single markdown document you write once and update occasionally, while the potential upside is being clearly understood by the systems deciding whether to cite you. Even if adoption is still early, the downside of having a clean llms.txt file is nothing, and the downside of an AI reader misunderstanding your site because it had to guess is real. That lopsided bet is why the file is worth creating now rather than waiting to see if it becomes universal.
What an llms.txt file actually is
An llms.txt file is a curated, human-written index of your site aimed at large language models, not a technical config file. It lives at the root, reachable at /llms.txt, and it is written in ordinary markdown so both a machine and a person can read it. Inside, you list the pages that matter most, grouped and labeled, each with a short description and a link. Think of it as the note you would leave for a smart assistant who just landed on your site and asked, in effect, what should I read first and why.
What it is not is a replacement for anything you already have. Your sitemap still lists every URL for crawlers, your robots file still governs access, and your pages still carry their own content. The llms.txt file adds a layer none of those provide: editorial judgment about what matters, expressed in language a model can read directly. A sitemap says here is everything. An llms.txt file says here is what is worth your attention and here is what each thing is, which is a fundamentally more useful message to send a reader that has to decide where to focus.
Where llms.txt came from
The convention was proposed in September 2024 by Jeremy Howard, co-founder of Answer.AI and a well-known figure in the machine learning world through his earlier work at fast.ai. The problem he named was concrete: the pages of a modern website are cluttered with navigation, scripts, and markup that make it hard for a language model to extract the actual content, and a model working within a limited context window benefits from a clean, curated pointer to what matters instead of the raw, noisy site. The llms.txt file was his proposed answer, a simple standard anyone could adopt without permission.
Understanding the origin matters because it explains the design. The file is deliberately simple, markdown so it is trivial to write and parse, curated so it respects the model’s limited attention, and placed at a predictable root path so it can be found the same way everywhere. It was not handed down by a search engine as a ranking mechanism, which means nobody is obligated to read it, but also that anybody can adopt it freely. That grassroots, low-friction character is exactly why it spread as a practice worth trying, even before any single AI company committed to consuming it universally.
How do you write one?
You write an llms.txt file in markdown, starting with an H1 that names your site, an optional short summary, and then sections of curated links with descriptions. A minimal version looks like this:
# Instant Press
> Press, publication placement, and AI-search visibility services.
## Core pages
- [Services](https://example.com/services): what we offer and who it is for
- [How AEO works](https://example.com/aeo): our approach to AI search visibility
- [Case studies](https://example.com/results): client outcomes with specifics
## Learn
- [AEO glossary](https://example.com/glossary): definitions of key terms
- [Blog](https://example.com/blog): guides on AI search and PRThat is the entire pattern: a heading, a one-line summary, and grouped links with plain descriptions of what each page is. Keep the descriptions honest and specific, because their whole job is to tell a model what it will find before it follows the link. Save the file as llms.txt, upload it to your site root so it resolves at /llms.txt, and confirm it loads in a browser. On a static host this is as simple as dropping the file in your public directory. There is no build step and no special server configuration required.
Write descriptions that do real work
The links in an llms.txt file are only half of it. The descriptions beside them are where the file earns its value, because a description is your one chance to tell a model what a page is and why it matters before the model decides whether to read it. A description that just repeats the page title wastes that chance. A description that states, in a plain clause, what the page covers and who it is for gives the model the context it needs to judge relevance. Treat each description as a tiny brief: specific, honest, and focused on what a reader will actually find.
The discipline pays off beyond the file itself, because writing these descriptions forces you to articulate what each of your important pages is genuinely about. If you struggle to describe a page in one clear clause, that is usually a sign the page itself is unfocused, which is worth knowing. Keep the descriptions short enough to scan and specific enough to be useful, avoid marketing language that a model learns to discount, and make sure each one accurately previews its page. A file full of sharp, honest descriptions reads like a knowledgeable guide to your site. A file of vague or inflated ones reads like noise, and a model treats it accordingly.
What to put in it, and what to leave out
Put in the pages you most want an AI to understand and cite: your core service or product pages, your best explanatory content, your definitional and reference pages, and anything that represents your expertise clearly. Curate ruthlessly. The value of the file comes from it being short and high-signal, so a model reading it gets your priorities immediately rather than drowning in every URL you have. If you list everything, you have just rebuilt your sitemap and thrown away the one advantage llms.txt offers, which is editorial focus.
Leave out the clutter: thin pages, duplicate content, utility pages, and anything you would not want to represent you in an answer. Also keep the descriptions truthful, because a description that oversells a page teaches a model to distrust your file, and a distrusted file is worse than no file. The discipline of writing llms.txt is the discipline of deciding what your site is actually for, which is a useful exercise regardless of who reads the result. Update it when your important pages change, and otherwise leave it stable so it stays a reliable map.
How llms.txt fits with your other files
It helps to place llms.txt correctly among the files it lives beside, because people confuse them. Robots.txt controls access: it tells crawlers what they may and may not fetch. Your XML sitemap enumerates every URL for discovery. Llms.txt does neither of those. It curates and recommends, offering a short, opinionated guide to your most important content in language a model reads directly. The three are complementary, not competing, and a healthy site can carry all of them, each doing its own job. Llms.txt is the only one of the three that expresses editorial judgment about what matters.
The convention also includes an optional companion, llms-full.txt, meant to hold a fuller, expanded version of your key content in one place rather than just links to it. The idea is that a model working within a limited context window can pull a single clean document instead of crawling many pages. Whether you add the companion depends on your site: for a small, content-dense site it can be worth providing, while for a large site it is often impractical to maintain. Start with the basic llms.txt file, which is the part that carries most of the value, and only add the fuller companion if you can keep it accurate. A stale llms-full.txt is worse than none, because it hands a model an outdated version of your content to repeat.
The mistakes that make an llms.txt file useless
The fastest way to waste the effort is to dump every URL on your site into the file, turning it back into a sitemap and discarding the one advantage it offers. The value is curation, so a bloated, uncurated llms.txt fails at the only job it had. List the pages that genuinely represent your best content and priorities, and leave everything else out. If a model reading your file cannot tell in a glance what your site is actually for, the file is not doing its work, no matter how many links it contains.
The other failures are quieter but just as damaging. Descriptions that oversell a page teach a model to distrust the whole file. Links that rot because you moved pages without updating the file send a reader to dead ends. A file that was accurate at launch and never touched again slowly drifts out of sync with the site it describes. Treat llms.txt as a small living document: keep the links valid, keep the descriptions honest, and revisit it when your important pages change. It asks very little maintenance, but it does ask some, and a neglected file quietly becomes a liability instead of the clean map you meant it to be.
Does llms.txt actually do anything yet?
Here is the honest part, and it matters: adoption is still emerging, and no major AI engine has publicly guaranteed that it reads and acts on llms.txt in a way that changes your visibility. Treat any claim that llms.txt is a proven ranking factor with suspicion, because the evidence for that does not exist yet. What is true is that the convention is spreading, that it costs almost nothing to implement, and that it aligns with the direction AI search is moving, toward reading curated, clean content over messy raw pages. That makes it a reasonable low-cost bet, not a guaranteed win.
So set your expectations correctly. Do not create an llms.txt file expecting an overnight jump in citations, because that is not what it is for and not what it will do. Create it because it is cheap insurance against being misunderstood, because it forces you to clarify what your best content is, and because if the practice does become widely consumed, you will already be set up for it. The realistic framing is a small, sensible investment in a growing convention, not a magic switch, and framing it that way protects you from the overhyped advice already circulating about it.
Add it now because the cost is near zero
The case for creating an llms.txt file today comes down to the lopsided math. It takes an afternoon, it requires no ongoing maintenance beyond occasional updates, and it carries no downside, while the upside, being clearly understood by an increasingly important class of readers, is real even if unguaranteed. Very few AI-visibility tactics are that cheap and that harmless. When something costs almost nothing and might help meaningfully, the rational move is to do it and move on, not to debate it for weeks.
Write the file, curate it to your genuinely important pages, describe each one honestly, and ship it to your root. Then return to the work that has a surer payoff, clear answer-first pages and credible outside coverage, knowing your llms.txt file is quietly in place if and when the engines lean on it more. To create an llms.txt file well, keep it short, keep it honest, and keep it focused on the content you would most want an AI to read and repeat. Do it once this week and you have covered a base most of your competitors have not even noticed yet.