What is llms.txt?
Short answer
llms.txt is a Markdown text file published at the root of a website (/llms.txt) to give large language models and AI agents a summary of the site and an annotated list of its key pages. It is an open proposal from September 2024, not a standard, and no major search engine currently confirms using it.
To understand what llms.txt is, it helps to go back to the source. Jeremy Howard, co-founder of Answer.AI, published the idea on 3 September 2024 in “/llms.txt—a proposal to provide information to help LLMs use websites”. The specification lives at llmstxt.org, last revised on 10 August 2026. People often search for it as llms txt, without the dot, but the file name never changes: lower case, with the dot.
The problem it sets out to solve is a practical one. A web page is built for people: menus, banners, scripts, ads. Turning that HTML into clean text is, in the proposal’s own words, “both difficult and imprecise”, and model context windows are still too small to take in an entire site. An llms.txt file offers a shortcut: a short Markdown index of what matters.
It is just as important to be clear about what it is not. It does not control bot access (that is the job of robots.txt, covered in my guide to robots.txt for AI), it does not opt your content in or out of model training and it is not a ranking signal. It is designed for the moment of the query: when a user or an agent needs to understand your website right now.
What does an llms.txt file look like? Format and llms-full.txt
The format is Markdown with a fixed structure, and almost all of it is optional. According to the specification, the file contains the following, in this order:
- An H1 with the name of the project or site (
# Name). This is the only required section. - A blockquote (
> …) with a short summary containing the key information needed to make sense of the rest. - Zero or more blocks of text (paragraphs or lists, but no headings) with further detail or instructions for use.
- Zero or more H2 sections (
## Title) containing “file lists”: each item is a Markdown link[name](url), optionally followed by:and a note. - A
## Optionalsection, by convention, for secondary links that an agent can skip when it needs a shorter context.
# Example Accountants
> Tax, payroll and bookkeeping for sole traders and small businesses in Cardiff. In person and online, in English and Welsh.
Key facts: founded in 2009, 12 staff, fees published on our pricing page. We do not take on statutory audits.
## Services
- [Tax for sole traders](https://www.example.com/services/tax.md): Self Assessment, VAT returns and Making Tax Digital
- [Payroll](https://www.example.com/services/payroll.md): payslips, contracts, PAYE and auto-enrolment
## Company
- [Pricing](https://www.example.com/pricing.md): monthly fees by type of client
- [Contact and opening hours](https://www.example.com/contact.md): address, phone number and office hours
## Optional
- [Blog](https://www.example.com/blog.md): tax and employment updates explainedMarkdown versions of each page
The proposal goes a step further: pages that are useful to an agent should offer a clean Markdown version at the same URL with .md appended (page.html.md) or with the extension replaced (page.md). For URLs without a file name, you append index.html.md or index.md. The current revision also explains how to advertise them: rel="alternate" type="text/markdown" for the Markdown version and rel="describedby" for the llms.txt file that covers it, using either a <link> element or an HTTP Link header.
You can publish the file at the root (/llms.txt) or in a sub-path such as /docs/llms.txt, in which case it covers only the pages beneath it.
What about llms-full.txt?
llms-full.txt does not appear in the llmstxt.org specification. It is a variant that Mintlify developed together with Anthropic: instead of an index of links, it contains the full text of your documentation in a single file, so that a model or a coding assistant can load it all at once. Anthropic, for example, links its llms-full.txt from its llms.txt.
Does your business need an llms-full.txt?
Hardly ever. It makes sense when you publish extensive technical documentation (an API, software, manuals) that people paste into Cursor, Claude Code or ChatGPT. For a 20-page company website, a standard llms.txt is enough.
Do ChatGPT, Google or Claude read llms.txt?
The straight answer, as of 9 October 2026: no major search engine or assistant has confirmed that it uses other websites’ llms.txt files to decide what to cite or how to rank. Google has ruled it out in writing for Search. Here is what we know, with dates.
- 17 June 2025. John Mueller writes on Bluesky: “FWIW no AI system currently uses llms.txt”, as reported by Search Engine Roundtable.
- 23 July 2025. Gary Illyes, speaking at Search Central Deep Dive Asia-Pacific, says that Google does not support llms.txt and has no plans to (Search Engine Land).
- May 2026. Chrome adds an experimental “Agentic Browsing” category to Lighthouse, including an llms.txt audit. It only checks that the file is served without server errors; a 404 is marked “Not Applicable”, because providing the file is “optional at the moment”.
- June 2026. Asked about this apparent contradiction, Mueller calls it “purely speculative for now” and advises creating the file when an AI platform that sends you customers asks for it (Search Engine Journal). A couple of weeks later, on the Search Off the Record podcast, he adds that it cannot help to tell one website from another, although it may help an automated system that is already on your site (SEJ).
- Today. Google’s official guide to generative AI features (updated 10 July 2026) lists “LLMS.txt files” among the things you can ignore: you do not need AI text files or Markdown to appear in Google Search, because Search ignores them. Creating them for other services, it says, does no harm.
OpenAI, Anthropic and Perplexity
All three publish an llms.txt for their own developer documentation (OpenAI, Anthropic, Perplexity). Yet none of them has documented that ChatGPT, Claude or Perplexity reads that file on other websites to choose its sources. Their guidance for webmasters talks about robots.txt, which is what their crawlers respect. Publishing a file for your developers is not the same as using it as a search signal.
What the server logs say
Studies based on real data point the same way:
- SE Ranking (November 2025, around 300,000 domains): 10.13% had an llms.txt file, and the study found no statistical relationship between having one and being cited by AI (SEJ).
- OtterlyAI (February 2026, 90 days, one test site): 84 requests for
/llms.txtout of more than 62,100 AI bot visits, roughly 0.1% (OtterlyAI). - Ahrefs (June 2026, 137,210 domains, May 2026 traffic): 28% publish llms.txt, but 97% of those files did not receive a single request during the month. Of the remaining 3%, only 19.5% of requests came from AI bots; Ahrefs concludes that the clearest audience is agents and coding tools such as Claude Code (Ahrefs).
- EZY.ai (July 2026, 83 websites, 12 weeks; a company that sells AEO tools): GPTBot requested llms.txt 7 times against 3,990 requests for robots.txt; ClaudeBot, 9 against 3,120; PerplexityBot, not once (EZY.ai).
28%
of the domains analysed publish llms.txt
Ahrefs, 137,210 domains, May 2026
97%
of those files received no requests in a month
Ahrefs, June 2026
84
requests for /llms.txt out of more than 62,100 AI bot visits in 90 days
OtterlyAI, February 2026
The honest conclusion
As of October 2026, llms.txt does not move your visibility in Google, ChatGPT, Claude or Perplexity. And a bot fetching the file does not mean it uses it: as Ahrefs puts it, “fetched” doesn’t mean “read”.
llms.txt SEO: does it actually help?
The relationship between llms.txt and SEO is quickly summed up: for rankings, it does nothing. For other things it does help, which is why it is worth about 30 minutes if your website has content that someone will want to use with an AI tool. These are the cases where it adds real value:
- Agents and coding tools. Claude Code, Cursor and similar assistants fetch an llms.txt when the user tells them to or when they browse documentation. This is the use the Ahrefs data confirms.
- Documentation and technical products. If you have an API, software or manuals, an index like this (and perhaps an llms-full.txt) makes it easier for your customers to work with your product from their assistant.
- Users who paste your website into a chat. A customer who gives ChatGPT or Claude the URL of your llms.txt hands it an accurate summary of your business, rather than leaving the model to interpret your HTML.
- Putting your own house in order. Writing it forces you to decide which 15 pages really matter and to sum up your business in one verifiable sentence. That same work improves your content for AI SEO.
It has costs too. If your pages change and the file does not, you are publishing an out-of-date version of your business. Ahrefs also flags the risk of prompt injection: this is a file designed to be read by models, so treat it like code and check who can edit it.
Key points
- llms.txt is a Markdown index for AI, proposed in September 2024. It is not a standard.
- Google states in writing that Search does not use it. OpenAI, Anthropic and Perplexity have not confirmed using it.
- Server logs show that the major AI bots almost never request it; the ones that use it are agents and coding tools.
- It is worth 30 minutes if you have documentation or want an accurate summary of your business. It does not deserve budget as an SEO tactic.
- robots.txt, clear content and structured data all come before llms.txt.
llms.txt vs robots.txt vs sitemap.xml
All three live at the root of the domain and are often confused, but they do different jobs. Only robots.txt decides what AI bots can crawl; if you want to fine-tune it, see my guide to robots.txt for AI.
| Criterion | llms.txt | robots.txt | sitemap.xml |
|---|---|---|---|
| Purpose | Summarise the website and point models and agents to its key pages | Tell crawlers which URLs they may crawl | List URLs so that search engines can discover them |
| Who reads it | Agents and coding tools when instructed to; no major search engine confirmed | Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and almost every legitimate bot | Google, Bing and other search engines |
| Format | Markdown: H1, blockquote and H2 sections with lists of links | Plain text with User-agent, Allow and Disallow directives | XML following the sitemaps.org protocol |
| Standard | Community proposal (llmstxt.org, 2024) | IETF RFC 9309 (2022) | sitemaps.org protocol 0.9 |
| Effect on rankings | None proven; Google says it does not use it | Indirect: what you block does not get crawled | Indirect: helps URLs get discovered, not ranked |
| Required | No | No, but almost universal | No; strongly recommended for large sites |
For a small business, the order of priority is clear: first, a robots.txt that does not accidentally block the bots you want; then a correct sitemap and content that answers real questions (I explain this in what is GEO and how it differs from SEO and AEO); and finally, if it suits you, llms.txt.
Free llms.txt generator
If you came here looking for an llms txt generator, here it is: this llms.txt generator is the quickest way to create a file that follows the specification. Fill in the fields with your website’s details. It comes preloaded with a fictional vineyard so you can see the result, and the URLs use example.com, a domain reserved for examples.
Before you upload it
Replace every URL with your own and open them one by one. Only switch on “Link to .md versions” if your website genuinely serves those Markdown versions; otherwise the links will return a 404.
How to create an llms.txt file in 6 steps
Writing an llms.txt by hand takes less than an hour for a company website. These are the steps, in order:
Choose the pages that matter
Draw up a short list of 10 to 30 URLs covering what an agent would need to understand and recommend your business: services or products, pricing, contact details, who you are and your most useful guides or product pages. Leave out legal pages, tag archives and pagination.
Write the H1 and the summary
Start with a # line containing the name of the site or company, which is the only required element. Below it, add a blockquote (>) of one or two sentences saying what you do, for whom and where, with verifiable facts and no slogans.
Group the links into H2 sections
Create ## sections (for example Services, Company, Guides) and, in each one, a list of links in the format - [Name](absolute URL): short note. The note explains what the model will find on that page.
Put secondary links under ‘Optional’
Add a ## Optional section at the end with the links that can be left out, such as the blog or news. By convention, an agent can skip that section when it needs a shorter context.
Publish it at the root of your domain
Upload the file as /llms.txt so that it returns a 200 status code, in UTF-8, as text/plain or text/markdown. In Next.js it goes in the public folder; in WordPress a plugin such as Yoast SEO or AIOSEO can generate it. Do not block it in robots.txt.
Validate it, link to it and keep it current
Check that it loads, that every link responds and that the Lighthouse llms.txt audit reports no error. Link to it from your site footer, check your server logs to see who requests it and update it whenever the pages it lists change.
If you want to see a real file, the one for this website is at josegalan.dev/llms.txt: an H1, a summary of what I do and the service and project pages. I maintain it alongside the rest of the site, because a file that contradicts the website costs you credibility instead of building it.
Common mistakes
Relative URLs instead of absolute ones, links to pages that no longer exist, a summary full of slogans, hundreds of links with no selection, and a file served as HTML or behind redirects. Any one of these cancels out the little the file can contribute.
Frequently asked questions about llms.txt
What is llms.txt?
llms.txt is a Markdown text file published at the root of a website (/llms.txt) that gives large language models and AI agents a summary of the site and an annotated list of its key pages. It was proposed by Jeremy Howard of Answer.AI on 3 September 2024. It is an open proposal, not an official standard.
Is there any llms.txt SEO benefit?
No. Using llms.txt for SEO has no proven effect, on Google or on ChatGPT. Google says in its guide to generative AI features (updated in July 2026) that Search ignores the file, and neither OpenAI, Anthropic nor Perplexity has documented that their search products take it into account. Studies by SE Ranking and Ahrefs find no link between having the file and being cited more often.
What is the difference between llms.txt and llms-full.txt?
llms.txt is an index: a title, a summary and links to the important pages. llms-full.txt, by contrast, gathers the full content of those pages into a single Markdown file so that a model can load it in one go. It is not part of the original specification: Mintlify and Anthropic popularised it for technical documentation. A small company website rarely needs one.
Does llms.txt replace robots.txt?
No. robots.txt tells crawlers which URLs they may crawl, and it is what GPTBot, ClaudeBot and PerplexityBot respect. llms.txt only suggests which content matters and grants or removes no permissions: if you block a bot in robots.txt, having an index for AI will not unblock it. Check robots.txt first; everything else can wait.
Where does the llms.txt file go?
At the root of your domain, so that https://yourdomain.co.uk/llms.txt returns a 200 status code, in UTF-8, as plain text or Markdown. The specification also allows a sub-path such as /docs/llms.txt to cover just that section. In Next.js, simply put it in the public folder; in WordPress, Yoast SEO or AIOSEO can generate it.
How often should you update your llms.txt file?
Whenever the pages it links to change: a new service, a URL that disappears or a price that goes up. An llms.txt with broken links or details that contradict your website is worse than having none. In practice, review it with every significant change to the site or, better still, generate it automatically from your CMS or on every deployment.
Sources
- Jeremy Howard, /llms.txt—a proposal to provide information to help LLMs use websites, Answer.AI, 3 September 2024.
- The llms.txt specification, llmstxt.org (revised 10 August 2026).
- Google Search Central, Guide to Optimizing for Generative AI Features on Google Search (updated 10 July 2026).
- Chrome for Developers, Lighthouse: llms.txt and Agentic Browsing scoring (5 May 2026).
- Search Engine Roundtable, Google: No AI System Currently Uses LLMs.txt, 18 June 2025.
- Search Engine Land, Google says normal SEO works for ranking in AI Overviews and llms.txt won't be used, 24 July 2025.
- Search Engine Journal, Google says llms.txt is purely speculative for now, 2 June 2026, and Google's Mueller says llms.txt can't help LLMs differentiate sites, 15 June 2026.
- Search Engine Journal, llms.txt shows no clear effect on AI citations (SE Ranking, 300K domains), 20 November 2025.
- OtterlyAI, The llms.txt experiment, 5 February 2026.
- Ahrefs, We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read, 15 June 2026.
- EZY.ai, We put llms.txt on 83 websites, 27 July 2026.
- Mintlify, What is llms.txt? (origin of llms-full.txt), updated 14 March 2026.
This article was created with the help of AI and reviewed by José Galán. I take great care over every post and every translation, but the odd mistake can still slip through. If you find one, write to me: you will be helping me improve.
Search & AI Visibility
Want to know how AI sees you, beyond a single file?
I review your robots.txt, your structured data and how ChatGPT, Perplexity and Google AI Overviews cite you, with a baseline you can repeat and measure.
See the Search & AI Visibility service