llms.txt is a plain-text, Markdown-formatted file placed at a website's root, at /llms.txt, that gives large language models a curated map to a site's most useful content. It lists the pages you most want a model to read, written in clean Markdown instead of cluttered HTML.
The file is a proposed community convention, not a ratified web standard. According to the original llms.txt proposal, llms.txt was proposed on September 3, 2024 by Jeremy Howard of Answer.AI, and its only required section is an H1 heading, followed by an optional blockquote summary and H2 file lists.
One point matters more than any other here. llms.txt is advisory: it controls nothing and blocks nothing. It cannot force a model to read your pages, and it cannot stop one from crawling anything.
Why Does llms.txt Exist? The Problem It Solves
llms.txt exists because web pages are built for people, not models. A typical page wraps its real content in navigation menus, ads, cookie banners, and JavaScript that add noise a model has to wade through.
That noise is expensive in two ways. Context windows still cannot hold a whole site at once, so a model can only read a slice of your content per request. Every token spent on menus and ads is a token not spent on your actual answer.
The cost is easy to underestimate. By Olostep's own observation, a single webpage's HTML might consume around 50,000 tokens, while the same content in Markdown uses only about 5,000 tokens. Cleaner input in AI-ready data formats lets a model spend its limited context on meaning rather than markup.
Why Markdown Instead of HTML or XML
Markdown wins because it removes boilerplate while keeping structure. It strips navigation, ads, popups, scripts, and cookie banners, yet preserves headings, lists, tables, and links that carry meaning.
Markdown also mirrors natural language, so a model parses its structure with fewer tokens than nested HTML tags or XML. This is why llms.txt uses Markdown rather than the site's raw source.
The catch is that converting HTML to Markdown can be lossy. A careless conversion drops tables, mangles code blocks, or leaves fragments of navigation behind. Clean conversion is what makes the resulting content actually useful to a model, and it is harder than it looks.
What Does an llms.txt File Look Like?
An llms.txt file is a short Markdown document with a fixed shape. It starts with an H1 project name, which is the only required part, followed by an optional blockquote summary, then H2 sections that hold Markdown "file lists" of links.
Each list item is a link to a page plus a short note describing it. Here is a minimal, copyable example:
# Acme Docs
> Acme is an API for sending transactional email. This file points language
> models to the documentation most useful for answering developer questions.
## Docs
- [Quickstart](https://acme.dev/docs/quickstart.md): Install the SDK and send your first email
- [Authentication](https://acme.dev/docs/auth.md): API keys, OAuth, and scopes
- [API Reference](https://acme.dev/docs/api.md): Every endpoint and parameter
## Optional
- [Changelog](https://acme.dev/docs/changelog.md): Recent releases and breaking changesRead this as a content map, not an access rule. The links point a model toward good pages; they do not allow or deny anything, and no line in the file can grant or revoke crawler access.
llms.txt vs llms-full.txt
llms-full.txt is a single file that contains the full Markdown content of your docs, not just links to them. It lets someone paste an entire documentation set into a model in one step, which is handy for coding assistants that need complete context.
The trade-off is size. A full-content file for a large site can run past practical token and file-size limits, which makes it slow to fetch and too big for smaller context windows. llms.txt stays small because it holds links; llms-full.txt grows with your content because it holds the content itself.
llms.txt vs robots.txt vs sitemap.xml
These three files sit at a site's root and are often confused, but they do different jobs. The table below compares them across the same criteria so the differences are clear.
| Criterion | llms.txt | robots.txt | sitemap.xml |
|---|---|---|---|
| Primary purpose | Curate key content for on-demand LLM use | Request which paths crawlers may access | List URLs for search indexing |
| Format | Markdown | Plain-text directives | XML |
| Audience | Language models and AI tools | Web and AI crawlers | Search engine crawlers |
| Enforcement | None; advisory only | Advisory; widely but not universally obeyed | None; a hint for indexing |
| Status | Community proposal (2024) | Long-standing convention | Supported standard |
The key correction is about control. robots.txt is a signal that well-behaved crawlers choose to respect, not a technical shield that blocks access. llms.txt is weaker still: it issues no allow or deny directive at all and only suggests what to read.
If you want to understand how bots reach your pages in the first place, our guide to web crawling and scraping covers the mechanics in depth.
How AI Crawlers Actually Read Your Website
Most AI systems today read your actual pages, not your llms.txt file. This is where the theory behind llms.txt meets how crawling works in practice.
The real pipeline runs in stages. An AI crawler such as GPTBot fetches a URL, may render its JavaScript to build the final page, strips the boilerplate, converts the HTML to Markdown or plain text, and then chunks that text to fit a context window. Nothing in that sequence requires an llms.txt file to exist.
Crawler traffic is also growing fast, which is part of why the topic gets attention. Per Cloudflare's crawler analysis, AI and search crawler traffic on Cloudflare's network grew 18% year over year from May 2024 to May 2025, and GPTBot's raw requests rose 305% over the same window.
How AI Answers Use Live Content (RAG and Grounding)
Retrieval-augmented generation, or RAG, is how many AI answers stay current: the system fetches live content at answer time and grounds its response on that text. It retrieves relevant pages, chunks them, feeds the chunks to the model, and keeps the source URLs so it can cite them.
This is why content quality drives citations more than a curated file does. A model grounds on the clean content for RAG it can actually fetch and parse at inference time. If your pages are hard to fetch or messy to convert, a tidy llms.txt does not fix the underlying problem.
Do AI Systems Actually Use llms.txt?
As of 2026, adoption of llms.txt is rising, but measured usage remains minimal. Publishing a file and having an AI system read it are two different things, and the evidence points to a large gap between them.
The strongest data comes from Ahrefs' 137K-site study: in a May 2026 analysis of 137,000 Ahrefs Web Analytics domains, 28% published an llms.txt file, yet 97% of valid files received zero requests that month. That 28% sample skews toward technical sites, so treat it as an upper bound rather than a figure for the whole web.
The mechanism explains the numbers. A file sitting at your root does not put your site on any AI system's radar, because nothing announces it and few crawlers request it. If no tool fetches /llms.txt, the file has no effect regardless of how well you write it.
What Google and Other AI Platforms Say
Google says its search does not use llms.txt at all. Its Google Search Central guidance, updated July 2026, states that you don't need special AI text files because Google Search doesn't use them and "ignores them."
A named Google engineer said the same about AI systems broadly. In John Mueller's public remark from June 2025, he stated that "no AI system currently uses llms.txt."
One nuance is worth separating out. OpenAI, Anthropic, and Google publish llms.txt files for their own documentation, but that is a choice about their docs and is not evidence that their models read third-party llms.txt files at inference time.
Should You Create an llms.txt File?
Create one if you run a documentation-heavy site, and keep your expectations modest otherwise. The file is low-effort and harmless to add, and it genuinely helps coding assistants navigate large docs when a developer points a tool at it.
Set the SEO expectations honestly. There is no evidence today that llms.txt improves search rankings or increases AI-search citations, so it is a weak tactic to prioritize for visibility. Add it for docs usability, and deprioritize it as a growth lever.
Spend your first effort where crawlers actually spend theirs. Clean, well-structured, easily fetchable pages are what AI systems fetch and convert, so page quality returns more than a single hand-maintained file.
How to Create and Serve llms.txt (and LLM-Ready Content)
Creating an llms.txt file takes four steps. Write your key links in Markdown using the H1, blockquote, and H2 file-list structure, save the result as llms.txt, host it at your site root so it resolves at /llms.txt, then validate that it loads and that every link works.
The harder, more durable goal is serving clean, LLM-ready content for your whole site rather than curating one static file by hand. That means generating Markdown or JSON for any page on demand, which scales to thousands of URLs instead of a short list. Olostep returns LLM-ready Markdown by default with automatic boilerplate removal, stripping navigation, ads, popups, scripts, and cookie banners while preserving headings, lists, tables, and links.
Two Olostep tools support that server-side approach. The AI web scraper API converts live pages into Markdown, HTML, JSON, or text with JavaScript rendering, and the ready-to-use web parsers return structured fields when you need consistent schemas rather than free-form text.
The Bigger Picture: Building for the Web's Second User
AI agents are becoming a second user of the web, reading pages alongside people. Whether or not llms.txt becomes a formal standard, the durable strategy is the same: make your content machine-readable and reliably fetchable.
That principle outlasts any single file format. An automated web research agent gathers and grounds on live pages the way a person would, so the sites it can parse cleanly are the ones it can use. Investing in clean, structured, fetchable content prepares you for that reader no matter which conventions win.
Frequently Asked Questions
Is llms.txt an official standard?
No; llms.txt is a community proposal introduced in September 2024, and no standards body has ratified it.
Where do I put the llms.txt file?
Place it at your site root so it resolves at /llms.txt, though it can also live at a subpath such as /docs/llms.txt for a specific section.
Does llms.txt help SEO or Google rankings?
No; Google states that its search ignores the file, and there is no evidence it improves rankings.
Is llms.txt the same as robots.txt?
No; robots.txt requests which paths crawlers may access, while llms.txt issues no access directives and only suggests useful content to read.
Does ChatGPT or Perplexity read llms.txt?
No AI platform has publicly committed to reading third-party llms.txt files, and independent measurement shows real usage is minimal today.
What is the difference between llms.txt and llms-full.txt?
llms.txt lists links to your key pages, while llms-full.txt contains the full Markdown content of those pages in a single file.
