What Is llms.txt? How It Works, Benefits, and How to Create One

llms.txt is a plain-text file at a website's root (/llms.txt) that briefs AI tools on what the site contains. It gives large language models (LLMs), the AI behind ChatGPT and Claude, a short summary and links to the site's most useful pages.

The llms.txt proposal and specification states the idea directly: "We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content." Jeremy Howard of Answer.AI published it on September 3, 2024, as a community proposal, not an official web standard.

The Problem llms.txt Solves

llms.txt solves a space problem: web pages carry code and clutter that crowd out the content an LLM needs. Every LLM has a context window, which is the limited amount of text it can hold in view at one time. HTML pages fill that window with navigation menus, scripts, ads, and markup.

LLMs measure text in tokens, which are small chunks of text, often a word or part of a word. Markdown is a lightweight text format that marks headings with # and links with [text](url). The same content written in Markdown usually needs fewer tokens than its HTML version.

InfoQ on Cloudflare Markdown reports: "The company says a blog post that requires 16 180 tokens in HTML shrinks to about 3 150 tokens when converted to Markdown." That is one Cloudflare example from a single vendor-reported page, and results on other sites may differ.

llms.txt applies the same logic to a whole site, so a model reaches short, clean content first. The same principle is why Olostep turns web pages into AI-ready web data, such as Markdown or JSON, before LLMs use them.

Who Actually Reads llms.txt

The intended readers of llms.txt are AI agents: programs that use an LLM to take actions, such as opening web pages, to finish a task. AI coding tools such as Claude Code are one example.

Two terms explain when these readers show up. Training is when a model learns from a huge dataset before release. Inference is when a finished model answers a prompt, sometimes by fetching (downloading) live pages to do it.

The spec's author expected llms.txt to help mostly at inference time. He also notes that software documentation is where the file is used most, because coding agents follow it to find API references and tutorials.

A crawler is a bot that visits pages automatically, and AI crawlers that fetch pages on demand are increasing. The Cloudflare Radar 2025 Year in Review found that "AI 'user action' crawling increased by over 15x in 2025." User action crawling means bots fetching pages because a person asked a chatbot a question, so the figure covers AI fetches in general.

Olostep calls these agents the web's second user. Our article on the agentic web explains how agents browse and use sites on a person's behalf.

What an llms.txt File Looks Like

An llms.txt file is a short Markdown document with a fixed order: a title, a summary, optional notes, and lists of links. Only the title is required.

The example below is for a fictional shipping API called Parcelwise, and it follows the v2 spec.

# Parcelwise

> Parcelwise is a shipping API for online stores. It creates shipping labels, compares carrier rates, and tracks parcels.

Parcelwise has a REST API and SDKs for Python and Node.js. Every API request needs an API key from the Parcelwise dashboard.

## Docs

- [Quickstart](https://parcelwise.example/docs/quickstart.md): Create an API key and print a first label
- [API reference](https://parcelwise.example/docs/api-reference.md): Endpoints, parameters, and error codes
- [Webhooks](https://parcelwise.example/docs/webhooks.md): Tracking events and retry rules

## Product

- [Pricing](https://parcelwise.example/pricing.md): Plans, label fees, and usage limits
- [Supported carriers](https://parcelwise.example/carriers.md): Carriers and countries covered

## Policies

- [Terms of service](https://parcelwise.example/terms.md): Account and API usage terms
- [Privacy policy](https://parcelwise.example/privacy.md): What customer data Parcelwise stores

## Optional

- [Changelog](https://parcelwise.example/changelog.md): Release notes by API version
- [Blog](https://parcelwise.example/blog.md): Shipping guides and product news

Each part of the file has one job. This table maps every element in the spec to its syntax and purpose.

What Is llms-full.txt?

llms-full.txt is a companion file that holds the full text of a site's most important pages in one Markdown file, instead of links to them. Documentation platforms use it as a convention, but it is not part of the core llms.txt spec.

The trade-off is size. An agent gets everything in one fetch. But a large llms-full.txt can exceed the model's context window, so the agent may see only part of it.

The v2 spec suggests a lighter option: serve a Markdown copy of each page at the same URL plus .md, such as page.html.md. If your pages exist only as HTML, tools that convert pages to Markdown can produce these copies.

llms.txt vs robots.txt vs sitemap.xml

robots.txt controls crawler access, sitemap.xml lists the URLs a site wants indexed, and llms.txt gives LLMs a short guide that controls nothing. The three files sit side by side, and adding llms.txt does not change the other two.

One misconception needs correcting: llms.txt cannot block AI training. To opt out, add User-agent rules to robots.txt for AI crawlers such as GPTBot (OpenAI) or ClaudeBot (Anthropic).

Does llms.txt Actually Work? What the Evidence Shows

There is no public evidence that llms.txt improves AI search visibility or citations. It is still cheap to publish, and it may help agents that read developer docs or product pages.

Keep one distinction in mind: a fetch means a bot downloaded the file, while a read means a model actually used it. Server logs can show fetches, but they cannot prove reading.

Many Sites Publish llms.txt, but Most Files Go Unread

Many sites in technical samples publish llms.txt, but most of those files get no requests. Ahrefs measured this across 137,210 domains in May 2026.

The Ahrefs 137K-domain llms.txt study reports that "28% of the 137K domains using Ahrefs Web Analytics publish an llms.txt file." Ahrefs calls 28% an upper bound, because its customers skew more technical and SEO-aware than the web as a whole.

Per Ahrefs' May 2026 study, "97% of those files received zero traffic in May 2026." AI bots also made zero requests for llms.txt files that did not exist, so they do not appear to go looking for one.

Among the roughly 3% of files that got any requests, named AI tools made about 19.5% of fetches. GPTBot ranked first and Claude Code ranked second, while SEO audit tools made 21.7% of all requests. Ahrefs treats every figure as a ceiling, since a fetch is not proof of use.

Google Search Ignores llms.txt

Google Search does not use llms.txt, and publishing one neither helps nor hurts your rankings there. Google's generative AI search guide states: "Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

That statement covers Google Search, including its generative AI features. It does not speak for ChatGPT, Claude, Perplexity, or coding agents.

Chrome offers a Lighthouse llms.txt audit as part of its Agentic Browsing checks. The audit is not a Google Search ranking factor.

For showing up in AI-generated answers, our guide to answer engine optimization (AEO) covers the page-level work involved.

When llms.txt Is Worth Creating

llms.txt is worth creating when AI agents or coding assistants are likely to act on your content. Use this list to decide by site type:

• API and developer docs: Coding agents follow llms.txt to find references and tutorials, according to the spec's author.
• SaaS products agents act on: The file gives an agent a direct path to pricing, limits, and setup pages.
• Libraries and SDKs: A developer can point an AI coding tool such as Claude Code at a docs URL, and the file gives it a starting point.
• Small local business sites: The file is low priority, unless your CMS creates it for free.

Do not treat llms.txt as a traffic source. The Cloudflare crawl-to-referral analysis found: "In June 2025, OpenAI's crawl-to-referral ratio was 1,700:1, Anthropic's 73,000:1."

A crawl-to-referral ratio counts pages crawled for every visitor sent back to the site. Cloudflare notes that native apps may not send referral data, which can inflate the ratios. The figures describe AI crawling in general.

How to Create llms.txt for Any Website

Creating llms.txt takes five steps, and the result is a valid file served at https://yourdomain.com/llms.txt. The steps fit most site types, though some hosted platforms limit access to the root folder.

Before you start, you need access to your site's root folder or CMS settings. You also need a list of the pages that matter most.

Step 1: Choose the Pages That Matter

Pick the 10 to 50 pages an agent needs, instead of copying every URL from your sitemap. Good candidates are docs, pricing, product pages, and policies.

Leave out tag pages, archives, and thin marketing pages. Every extra link takes up space in the agent's context window.

For large or unfamiliar sites, build a full URL inventory first, then choose from it. The Olostep Maps endpoint discovers a site's URLs, so you can pick from a complete list.

Step 2: Write the File in Markdown

Write the file in any text editor, in the spec's order: H1 title, blockquote summary, notes, then H2 link sections. Save it as llms.txt with UTF-8 encoding, the standard text encoding on the web.

Give each link a one-line, factual note about what the page contains. Link to canonical URLs, meaning the main public version of each page, without tracking codes or redirects.

Prefer pages that show their content without JavaScript, or link to their .md versions. An agent that downloads raw HTML without running scripts may see an empty page on a JavaScript-heavy site.

To save time, a free llms.txt generator can draft the file from your site name, homepage URL, description, and important links. Edit the draft before publishing, since you know which pages matter most.

Step 3: Publish It on Your Platform

Publish the file so it loads at /llms.txt on your domain, using the method your platform supports. Check your platform's current docs, because plugin and app features change between versions.

Step 4: Validate and Test the File

Validate the file by confirming it loads correctly, follows the format, and links to working pages. Run curl -I https://yourdomain.com/llms.txt in a terminal to see the status code and content type.

Work through this checklist:

• The URL /llms.txt returns HTTP status 200, the code for a successful request.
• The server sends a text/plain or text/markdown content type.
• The file is saved with UTF-8 encoding.
• The file has exactly one H1, on the first line.
• Every link uses [name](url) syntax.
• Every linked page loads without a login or a chain of redirects.
• Neither robots.txt nor bot protection, such as a firewall challenge page, blocks the file.

Next, run the spec's own test: give an AI agent only your llms.txt and ask it questions about your site. If it can't find your pricing or setup steps, fix or add those links.

Finally, search your server logs for requests to /llms.txt and note which user agents (bot names) appear. Based on the Ahrefs data above, expect few requests.

Step 5: Keep It in Sync With Your Site

Regenerate llms.txt whenever important pages are added, moved, or removed. Add this step to your CI/CD pipeline, the automated process that tests and deploys your code.

To automate it, crawl the site, pull titles and summaries from the Markdown output, and rebuild the file. Olostep's Crawls endpoint can crawl every public page in the section you choose and return each page's content as Markdown or HTML.

curl -X POST "https://api.olostep.com/v1/crawls" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "start_url": "https://parcelwise.example/docs",
    "max_pages": 100,
    "include_urls": ["/docs/**"]
  }'

include_urls takes a glob pattern, a wildcard rule where /docs/** matches every page under /docs/. max_pages caps how many pages the crawl visits.

The response returns a crawl id, which you use to retrieve the results. You can also pass a webhook to get a notification when the crawl finishes.

According to Olostep's crawl documentation, crawlers respect robots.txt rules by default and use headless browsers to render JavaScript. Client-rendered docs pages can still return content.

The last step turns crawl results into the file. The sketch below is pseudocode, and its field names are illustrative, not Olostep response fields.

# Pseudocode sketch: build llms.txt from crawl results
lines = ["# Parcelwise", "", "> Shipping API for online stores.", "", "## Docs", ""]
for each page in crawl results:
    title   = first H1 heading in the page's Markdown
    summary = first sentence of the page's Markdown
    add line "- [" + title + "](" + page URL + "): " + summary
write lines to public/llms.txt

Review the generated notes before each publish, because a page's first sentence is not always a good summary. A practical path is to draft the file with the generator, then switch to a scheduled crawl once the structure is stable.

Common llms.txt Mistakes to Avoid

These mistakes make llms.txt harder for agents to use or trust:

• Dumping the whole sitemap: Hundreds of links bury the pages that matter and use up the agent's context window.
• Linking JavaScript-only pages: Agents that skip scripts may see blank pages, so link rendered HTML or .md versions.
• Including broken or gated links: Pages that return errors or require a login give the agent nothing to read.
• Treating it as access control: llms.txt blocks nothing, so crawler rules belong in robots.txt.
• Writing marketing copy: Notes like "best-in-class platform" tell an agent nothing, so state what each page contains.
• Letting it go stale: Old links and outdated pricing can lead agents to give users wrong answers.
• Trusting unknown files blindly: Ahrefs flags prompt injection, where hidden instructions in a file try to steer an agent. Agent builders should treat any llms.txt as untrusted input.

Frequently Asked Questions About llms.txt

Teams new to llms.txt often ask about its status, limits, placement, and upkeep.

Is llms.txt an official standard?

No. llms.txt is a September 2024 community proposal, now in version 2, that no standards body such as the W3C or IETF has adopted.

Does llms.txt help SEO or AI citations?

There is no public evidence that llms.txt improves rankings or AI citations. Google Search ignores the file, and Ahrefs' May 2026 log data shows most files get no requests.

Can llms.txt block AI crawlers from training on my content?

No. To opt out of AI crawling, add robots.txt User-agent rules for bots such as GPTBot or ClaudeBot.

Where should llms.txt go?

Place it at your site root, at /llms.txt. The v2 spec also allows subpaths such as /docs/llms.txt, which apply to that section of the site.

Do I need both llms.txt and llms-full.txt?

No. Start with llms.txt, and add llms-full.txt only when agents need full docs content in a single fetch.

How often should I update llms.txt?

Update it whenever important pages are added, moved, or removed. Regenerating the file on every deploy keeps it from drifting out of date.