llms.txt
A Markdown file at a site's root that tells AI crawlers what the site is about and which URLs are canonical — a kind of robots.txt for language models.
The short answer
llms.txt is a Markdown file at a domain's root that tells language models what a site is for and which URLs are worth reading. It carries a site name, a one-sentence description, optional context, and curated link sections. It grants nothing and blocks nothing — it is a reading list, not an access-control file.
What llms.txt is, and how it differs from robots.txt and sitemap.xml
llms.txt was proposed by Jeremy Howard at Answer.AI in September 2024 and is defined in Markdown for a reason: the format models parse most reliably is the one they were trained on most. The file opens with an H1 site name, a blockquote holding one sentence of description, an optional paragraph of context, then H2 sections of links where each link carries a colon and a short gloss explaining what the page is for.
The comparison everyone reaches for is robots.txt, and it is the wrong one. robots.txt is an access-control file: it names user agents and tells them which paths they may not fetch, and a crawler that ignores it is misbehaving. llms.txt has no directives at all. It cannot allow or deny anything, and a model that ignores it has broken no rule. One constrains crawling; the other curates comprehension. A site can serve both, and should, because they answer different questions.
The subtler confusion is with sitemap.xml, because both list URLs. A sitemap is exhaustive, machine-oriented and ordered by nothing — it exists so a crawler can discover every page, including the four hundred near-identical ones. llms.txt is deliberately incomplete and editorial: ten to thirty URLs, each with a human-written gloss, arranged so a model reading the file once understands the shape of the product. A sitemap answers what exists. llms.txt answers what matters.
How llms.txt is measured
Presence is binary, but the grade is not. In the AIRank rubric a present and well-formed llms.txt is worth 3 of the 20 points in technical readiness, with llms-full.txt worth a further point when present or explicitly not applicable. Well-formed means all four structural elements resolve: an H1, a blockquote description, at least one H2 section, and links that carry a gloss after the colon. A file that is Markdown-shaped but missing the blockquote scores as partially formed, not as present.
Beyond the pass/fail, three numbers are worth tracking yourself. Link resolution rate is the share of listed URLs that return 200 — anything below total is a live defect, because a dead link in a curated list is worse than no list. Entry count wants to sit between ten and thirty; below that the file is not describing a product, above that it is a sitemap wearing a costume. And staleness, measured as the gap between your last content change and the file's last modification, is the number that quietly ruins otherwise good files.
You'll know it worked when
Your server logs show AI crawlers fetching /llms.txt on their own initiative rather than only after you request a scan, and the pages models cite start being the ones you listed rather than whichever tangential blog post happened to rank.
A worked example
Below is a complete file for a small SaaS product. Read it top to bottom the way a model does: the H1 fixes the entity name, the blockquote gives the one sentence that will be reused verbatim when something asks what the product is, and the free paragraph supplies the context a title tag cannot carry — in this case, that the product has three distinct surfaces.
# AIRank
> AIRank is an AI-visibility platform that tracks where ChatGPT, Claude,
> Perplexity and Google AI Mode cite your brand, audits your site against a
> 47-point AI-readiness rubric, and ships the fixes.
The product has three surfaces: a citation tracker, a scanner, and a copilot
that writes and publishes fixes. All three run against one connected site.
## Docs
- [Getting started](https://www.airank.tech/docs/getting-started): Account setup and first scan.
- [Running your first scan](https://www.airank.tech/docs/first-scan): What the 47 checks cover.
- [How citations work](https://www.airank.tech/docs/how-citations-work): Sampling and match methodology.
- [Understanding the AI Score](https://www.airank.tech/docs/understanding-ai-score): The rubric and its weights.
- [API reference](https://www.airank.tech/docs/api): REST endpoints.
## Integrations
- [WordPress](https://www.airank.tech/integrations/wordpress): Plugin install and schema injection.
- [Shopify](https://www.airank.tech/integrations/shopify): App install for product schema.
## Optional
- [Changelog](https://www.airank.tech/changelog): Weekly release notes.
- [Blog](https://www.airank.tech/blog): Long-form essays.Each link is followed by a colon and a gloss, and that gloss is doing most of the work. Without it a model sees ten titles and has to guess which one explains pricing versus methodology. The Optional heading is not decoration either — it is a documented signal that everything beneath it is lower priority, so a model under a context budget knows what to drop first. Nine entries here, all resolving, all glossed, is a better file than one listing your whole sitemap.
Three ways llms.txt files go wrong
- The sitemap dump. Someone wires the file to generate from the CMS and it emerges with four hundred URLs, no glosses, and every tag archive included. This is strictly worse than having no file, because you have now spent the model's one cheap pass at understanding your site on noise you chose. The fix is to hand-write it, cap it near thirty entries, and treat any automated generation as a draft a human edits before it ships.
- The stale manifest. The file gets written during launch week, links the docs structure that existed then, and is never touched again. Six months later a third of its URLs redirect and two of them 404, which teaches every crawler reading it that your curation is unreliable. The fix is to put the file in the same review cycle as your navigation — if a link changes in the nav, it changes here, and a scheduled link check catches the rest.
- The HTML llms.txt. A framework serves the route through the normal page pipeline, so the file arrives wrapped in a layout, or with a text/html content type, or rendered client-side. Models expect raw Markdown at that path and a wrapped file parses as nothing. The fix is to serve it as a static asset with a plain text content type, and to verify by fetching the URL with a bare HTTP client rather than looking at it in a browser.
llms.txt against the other files at your root
| File | What it optimises | How you measure it | Where it lives |
|---|---|---|---|
| llms.txt | A model's one-pass comprehension of the site | Well-formed structure, entry count, link resolution rate | Markdown at the domain root |
| llms-full.txt | Ingesting the full doc corpus in a single fetch | Byte size and coverage of the canonical docs set | Markdown at the domain root |
| robots.txt | Which agents may fetch which paths | Crawler access per user agent, checked in server logs | Plain text at the domain root |
| sitemap.xml | Discovery of every indexable URL | Submitted, discovered and indexed URL counts | XML at the root, referenced from robots.txt |
The relationship the table hides is that these files fail in sequence, not in parallel. If robots.txt blocks an AI crawler, nothing below it matters — your llms.txt will never be fetched and your sitemap will never be read. Once access is granted, the sitemap handles breadth and llms.txt handles priority, which is why serving one is never a reason to skip the other. llms-full.txt is a specialisation, not an upgrade: it pays off for documentation and API references and actively hurts for marketing pages, where inlining everything buries the signal you were trying to raise.
In AIRank
AIRank checks for a well-formed llms.txt on every scan. It's worth 3 points on the 47-point rubric, with a further point for a matching llms-full.txt. The WordPress plugin and Shopify app can both auto-generate and serve the file for you, kept in sync with your content.
- Generative Engine Optimization
The practice of making a website more likely to be cited by AI answer engines (ChatGPT, Claude, Perplexity, Google AI Mode) rather than simply ranked on a traditional search results page.
- Schema Markup
Structured data embedded in a page (usually as JSON-LD) that describes what the page is about in a machine-readable vocabulary defined at schema.org.
- Indexing API
An API — most commonly IndexNow (Bing/Yandex) or Google's Indexing API — that pushes URL changes to a search index in real time instead of waiting for crawlers.
Does llms.txt replace robots.txt?
No, and it cannot. robots.txt is the only one of the two that carries directives — it names user agents and denies paths, and blocking an AI crawler there makes everything else moot. llms.txt has no allow or deny semantics at all. Serve both: robots.txt to grant access to the crawlers you want, llms.txt to tell them what to read once they are in. Referencing the llms.txt path from robots.txt saves them a guess.
How many URLs should I put in llms.txt?
Ten to thirty for most sites. The file's whole value is that it is shorter and better curated than your sitemap, so every entry you add dilutes the ones already there. Pick the pages that explain what the product is, how it works, what it costs, and how to start. Push release notes, blog archives and anything seasonal under the Optional heading, which signals to a model that those can be dropped first.
Do I need llms-full.txt as well?
Only if you have a documentation corpus worth inlining. llms-full.txt repeats the structure of llms.txt but embeds the full text of every linked page, which turns it into hundreds of kilobytes. That trade is good for API references and getting-started guides, where a model ingesting everything in one fetch genuinely answers better. It is a bad trade for marketing pages and blog posts, where the added volume lowers the signal-to-noise of the whole file.
Which AI crawlers actually read llms.txt?
Support varies by vendor and changes, so treat any specific list as perishable. The honest framing is that the file is cheap to ship, carries no risk, and pays off the moment any retrieval layer in your stack honours it. Check your own server logs for fetches of the path by AI user agents — that is the only source of truth for your site, and it is more useful than a compatibility table someone published last quarter.
Why does my llms.txt not work when I open it in a browser?
Almost always because a framework is serving the path through the page pipeline rather than as a static asset, so it comes back wrapped in a layout or with a text/html content type. Fetch it with a plain HTTP client instead of a browser and look at the raw body and headers. You want unwrapped Markdown and a plain text content type. If a route handler is generating it, make sure it sets that header explicitly.
Does llms.txt affect Google rankings?
No. It is not a ranking signal and it is not part of any documented search algorithm. Its value sits entirely on the retrieval side: a model or grounding layer that reads it starts with a correct picture of what your site is and which pages are canonical, which changes what gets cited rather than what gets ranked. Treat any claim that it moves blue-link positions with suspicion.
Written by
The AIRank Editorial Team
Research & editorial, AIRank
The AIRank editorial team runs the 47-point scanner, the Observer pings, and the GEO research programme every week. Writing is reviewed by the core engineers who build the Injector, Blaster, and Surgeon agents.
Last reviewed
About the team →