AI discovery file generator

llms.txt Generator

Create a clear llms.txt file that tells AI agents what your site is, which pages matter, and where developers should start. Generate a clean draft in seconds.

llms.txt Generator

llms.txt

Input

Site or product name

Homepage URL

Description

Important links

Agent notes

Output

Run the tool to see output here.

AI-agent friendly
Docs-ready format
Editable output

Developer experience

Discover pages and generate llms.txt via API.

Use the Crawls endpoint to discover all public pages on a domain, then build an llms.txt file from the results.

REST APIPage discoveryFull-site crawlMarkdown outputConfigurable depthURL filtering

Read the docs

# 1. Start the crawl
curl --request POST \
  --url https://api.olostep.com/v1/crawls \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "start_url": "https://example.com",
    "max_pages": 50,
    "include_urls": ["/**"],
    "include_external": false
  }'

# 2. Poll until completed (replace CRAWL_ID)
curl --url https://api.olostep.com/v1/crawls/CRAWL_ID \
  --header 'Authorization: Bearer YOUR_API_KEY'

# 3. Get crawled pages
curl --url 'https://api.olostep.com/v1/crawls/CRAWL_ID/pages?formats=["markdown"]' \
  --header 'Authorization: Bearer YOUR_API_KEY'

A better entry point for AI agents

A good llms.txt file compresses your product, docs, and important URLs into a format assistants can scan quickly.

Clear summary
Generate a concise product description and guidance for how AI tools should use your site.
Important links
Add docs, API references, pricing, changelog, support, and other pages that help agents answer accurately.
Markdown format
The result is plain text Markdown you can place at /llms.txt and version with your site.

How it works

Each tool is designed around a direct job: provide the input, get a useful output, and move it into your workflow.

1
Describe your site
Add the product name, homepage, and a concise description.
2
Add key URLs
Include docs, API references, guides, pricing, and support pages.
3
Publish the file
Place the generated content at your domain root as /llms.txt.

Built for practical workflows

These tools are small on purpose: they solve the conversion job quickly, then point teams toward the API path when the workflow becomes recurring or high volume.

Developer tools
Point AI coding assistants to docs, API references, SDKs, and examples.
SaaS products
Summarize your product and expose pricing, support, and onboarding pages.
Documentation sites
Give agents a curated map of the pages that matter most.

Need to keep llms.txt fresh?

Use Olostep crawls and monitors to track your docs, changelog, and product pages, then regenerate AI-facing files when content changes.

Get API keys API docs

Frequently asked questions

Product & Capabilities

What is llms.txt?
llms.txt is a plain text file intended to give AI systems a concise, curated guide to a website's most useful content.
Where should I publish llms.txt?
Publish it at the root of your domain, for example https://example.com/llms.txt.
Is llms.txt the same as robots.txt?
No. robots.txt gives crawler access instructions. llms.txt gives AI systems curated context and links.

Usage & Automation

What format does this generator output?
It outputs plain text Markdown suitable for an llms.txt file.
Can I edit the generated file?
Yes. The output is a starting point and should be reviewed before publishing.
Should I include every URL?
No. Include high-signal pages such as docs, API references, guides, pricing, support, and changelog.

Pricing & Plans

Is the generator free?
Yes. The generator runs locally in the browser.
Do I need an Olostep account?
No account is needed to draft a file. Olostep is useful when you want to crawl and maintain source content at scale.
Can Olostep automate updates?
Yes. Olostep can monitor and extract site content that can feed an automated llms.txt workflow.

Olostep sells to AI agents - live prices and self-serve payment

llms.txt Generator