How to llms.txt.
llms.txt is a single Markdown file at your domain root that tells AI crawlers what your site is about and which pages matter. It's a 2024 proposal from Jeremy Howard / Answer.AI that's now honored by Claude, Perplexity, and Bing's grounding layer.
The short answer
Setting up llms.txt means serving one Markdown file from your apex path containing an H1 site name, a blockquote one-sentence description, an optional context paragraph, and 10–30 described URLs grouped under H2 headings with the lowest-priority set under `## Optional`. Reference it from robots.txt so crawlers stop guessing.
Before you start
The only hard requirement is the ability to serve a static file from your apex path. That sounds trivial and frequently is not: hosted website builders often only allow pages under a route prefix, and single-page-app deployments commonly rewrite every unmatched path to index.html, which produces a 200 response containing HTML for a file that does not exist. You also need edit rights on robots.txt, which on many stacks is generated rather than authored.
- Apex-path file serving — verified by content type, not by status code alone.
- Edit access to robots.txt, including the ability to inspect rule ordering.
- A decided list of 10–30 canonical URLs, which is an editorial call rather than a technical one.
- A deploy pipeline that will not strip or minify a .txt file on the way out.
Fifteen minutes covers writing and deploying the file. Budget roughly an hour on top for the argument about which URLs belong in it, because that argument is where the actual value is created. For a real team the calendar cost is usually one deploy cycle plus one review, and the most common blocker is nobody owning the domain root — it belongs to infrastructure, while the content belongs to marketing.
Why this works
The hard problem in retrieval is not finding your URL. It is understanding, in one pass, what your site is for, which page is the canonical explanation of each concept, and which pages are duplicative. A crawler reading your sitemap sees four hundred URLs with near-identical titles and no way to rank them by explanatory value. A crawler reading llms.txt sees ten URLs with a one-line description each, plus a prose intro that frames the whole thing.
That framing is what the ## Optional heading exploits. It is an explicit priority declaration — a hint to prefer the sections above it when answer quality matters — and there is no equivalent signal anywhere else in your site's structure. Support varies by crawler because llms.txt is a proposal from September 2024 rather than a ratified standard, so treat it as a cheap, broadly-honoured hint rather than a contract you can rely on.
A sitemap tells a crawler where everything is. llms.txt tells it which ten things are worth reading, and that is a strictly harder question to answer from the outside.
The steps
- 01
Create the file
Make a file called
llms.txt. It lives at your site root —https://yoursite.com/llms.txt— and it is plain Markdown, not HTML and not JSON. Start with an H1 that is your site or product name, nothing else on that line. The path matters more than people expect: a file at /docs/llms.txt or /pages/llms-txt is not discoverable, because crawlers check the apex path and nowhere else. If your host rewrites unknown paths to a single-page-app shell, fix that before writing a word of content.Check
You'll know it worked when curl -I https://yoursite.com/llms.txt returns 200 with a text/plain or text/markdown content type and no redirect to your 404 template.
- 02
Write the one-sentence description
Directly under the H1, write a Markdown blockquote with one sentence explaining what the site is. Example:
> AIRank tracks where ChatGPT, Claude, and Perplexity cite your brand, and ships automatic fixes.This is the highest-signal line in the file and the one most often omitted. Without it, models fall back to extracting your<title>tag, which is usually a truncated brand-plus-tagline string that explains nothing. Name the category, the mechanism and the audience in that one sentence. Resist adjectives — 'leading', 'powerful' and 'seamless' carry zero retrieval signal.Check
You'll know it worked when the first three lines of the file are an H1, a blank line, and a line beginning with a
>character. - 03
Add a context paragraph (optional but recommended)
One paragraph of prose explaining the product or site. No marketing language — this is not for humans, it is for retrieval, and the reader has no other context about you. State what the thing does, what its main surfaces are, and who uses it. Aim for 30–80 words. The useful test is whether somebody who has never heard of you could describe your category correctly after reading only this paragraph. If they would need the rest of the page to get there, the paragraph is not doing its job.
Check
You'll know it worked when the paragraph pasted alone into an empty text file still explains what the site does, with no surrounding page and no brand knowledge assumed.
- 04
List your primary URLs
Under an H2 heading such as
## Docs, list the URLs that best explain the product, one per line as[Title](https://...): one-line description. Aim for 10–30 URLs total across sections, grouped as Docs, Examples and Optional. The## Optionalheading is a real signal — everything under it is declared lower-priority, which lets a model prefer your main sections when answer quality matters. Every description should say what the page contains, not why it is great. This is not a sitemap, and treating it like one destroys its entire value.Check
You'll know it worked when every URL in the file returns 200 to an anonymous curl -sIL check with no redirects and no canonical tag pointing somewhere else.
- 05
Deploy and reference it in robots.txt
Put the file at your root so it is served at
/llms.txt, then add a line to yourrobots.txt:LLMs: /llms.txt. This helps crawlers discover it without guessing at the path. While you are in robots.txt, confirm you are not blocking the AI crawlers you want reading the file — a Disallow rule that predates all of this is a common and entirely silent cause of zero effect. Check the rule ordering too, since a broad Disallow earlier in the file can shadow a narrower Allow below it.Check
You'll know it worked when curl https://yoursite.com/robots.txt shows the
LLMs: /llms.txtline and no Disallow rule above it matches that path. - 06
Consider /llms-full.txt
For API docs and reference pages, add a companion
/llms-full.txtwith the full text of each listed URL inlined beneath its link. This is expensive — often hundreds of kilobytes — but it lets a model ingest your entire documentation corpus in a single fetch instead of crawling twenty pages and hoping. It is worth it for docs, getting-started guides and FAQ content. Skip it for blog posts and marketing pages, where the signal-to-noise ratio is worse than letting the crawler pick for itself. In the AIRank rubric it is worth one additional point.Check
You'll know it worked when /llms-full.txt returns 200 and its byte count is at least an order of magnitude larger than /llms.txt — identical sizes mean the page bodies were never inlined.
How to verify it worked
Check your access logs before you check anything else. The file typically shows up in crawler access patterns within about 48 hours of deployment, so grep your logs for the /llms.txt path and record which user agents fetched it and how often. No fetches after a week means a discovery problem — usually a robots.txt rule or a wrong path — and nothing downstream can possibly work until that is fixed.
The second-order check is qualitative and slower. Watch whether answers about your product start citing your own documentation instead of a third-party summary of it. That shift lags deployment by one to two weeks as retrieval signals re-index, and it shows up first on technical and documentation queries. Marketing-page impact is genuinely smaller here, so do not construct your measurement panel out of buying-intent queries alone.
Scope
llms.txt changes which of your pages get chosen far more than whether you get chosen at all. Judge it on citation quality, not citation count.
Three ways this goes wrong
- The SPA-fallback 200. Symptom: the URL loads in a browser and returns a 200, yet no crawler ever seems to read the file. Cause: your host rewrites every unmatched path to index.html, so /llms.txt returns your app shell with a text/html content type and a body full of JavaScript. Fix: check the content type and the first bytes of the response, not the status code. Add an explicit static route or an exclusion rule ahead of the catch-all rewrite, then re-test with curl rather than a browser.
- The sitemap-dump anti-pattern. Symptom: you generated the file automatically, it lists four hundred URLs, and nothing changed. Cause: you have reproduced the exact disambiguation problem the file exists to solve, in a different format — a model reading it is no better off than a model reading your sitemap. Fix: delete it and hand-pick ten to thirty URLs, each with a description of what the page contains. If choosing is hard, that difficulty is information about your information architecture.
- The stale-mirror drift. Symptom: the file has been live for months, and half the URLs in it now redirect or 404. Cause: it was written once and never wired into any process that notices when pages move. Fix: add a link check to CI that fails the build when any URL in llms.txt does not return 200, and regenerate the descriptions whenever the linked page's purpose changes. Broken links cost you precisely the pages you most wanted read.
What to expect
llms.txt shows up in crawler access patterns within about 48 hours of deployment. The citation impact lags by 1–2 weeks as the retrieval signals are re-indexed. The largest wins are on technical/documentation queries — marketing-page impact is smaller.
Does llms.txt need to be in my sitemap?
No. Crawlers that honour it look for it at the apex path directly, and a sitemap entry for a non-HTML file adds nothing. What does help discovery is the `LLMs: /llms.txt` line in robots.txt, which is the conventional pointer. If you want a belt-and-braces approach, linking it from your footer costs nothing, but the apex path is what actually gets checked.
Is llms.txt part of robots.txt or a separate file?
Separate. robots.txt controls crawl permission and uses its own directive syntax; llms.txt is Markdown and controls nothing — it describes. The only relationship is the `LLMs: /llms.txt` pointer line you add to robots.txt for discovery. Confusing them causes real problems: putting Disallow rules into llms.txt does nothing, and putting Markdown into robots.txt can invalidate the rules below it.
Do I need llms-full.txt if I already have llms.txt?
Only if you have documentation worth ingesting whole. The full variant inlines each linked page's complete text, which is enormous but lets a model absorb your entire doc corpus in one fetch instead of twenty crawls. It is worth it for API references and getting-started guides. For a marketing site it is a large file with a poor signal-to-noise ratio and it is worth one rubric point.
How many URLs should llms.txt contain?
Ten to thirty, grouped into two or three sections. Fewer than ten and you have probably omitted something a model needs to answer a common question about you. More than thirty and you are back to the sitemap problem, where every URL dilutes the signal from the others. The right test is whether a stranger could explain your product after reading only the titles and descriptions.
Will llms.txt stop AI crawlers from reading the rest of my site?
No. It has no exclusionary semantics at all — it is a description, not a permission file. Crawlers will continue fetching whatever robots.txt allows them to fetch. If you want to restrict access, that is a robots.txt job, and it is worth checking that an old Disallow rule is not already blocking the crawlers you are now trying to help.
Written by
The AIRank Editorial Team
Research & editorial, AIRank
The AIRank editorial team runs the 47-point scanner, the Observer pings, and the GEO research programme every week. Writing is reviewed by the core engineers who build the Injector, Blaster, and Surgeon agents.
Last reviewed
About the team →