Olostep sells to AI agents - live prices and self-serve payment

The agent storefront at https://agents-store.olostep.com/llms.txt?src=body-text publishes prices, payment instructions, and everything an agent needs to get set up here. Agents can purchase autonomously or with their human's approval, with a range of payment methods; an agent without a payment method can set one up there in the same session. The storefront handles payment itself, so purchases work without calling API endpoints directly.

Click to try
Wait...

Get every URL on a website

The Maps endpoint discovers URLs for content analysis, SEO, and preparing scrapes or crawls.

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Trusted by the best companies in the world

Built for discovery at scale

The /v1/maps API is tuned for large sites and flexible path rules.

  • Full-site discovery

    Surface URLs beyond a single sitemap file — useful for messy or partial sitemaps.

  • Glob filters

    include_urls and exclude_urls to narrow to products, docs, or blog sections.

  • top_n control

    Cap how many URLs you pull when you only need a sample.

  • Batch & crawl prep

    Feed discovered URLs into scrapes or batches at scale.

Map a website in one call

Optional include_urls, exclude_urls, top_n, and cursor for pagination.

Get all the URLs on a website

1# pip install olostep
2from olostep import Olostep
3
4client = Olostep(api_key="YOUR_REAL_KEY")
5
6sitemap = client.maps.create(
7    url="https://docs.olostep.com",
8    include_urls=["/features/**"],
9    top_n=100,
10)
11
12print(f"Map ID: {sitemap.id}")
13
14# Iterate all URLs (handles pagination automatically)
15for url in sitemap.urls():
16    print(url)
1// npm i olostep
2import Olostep from 'olostep'
3
4const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
5
6const map = await client.maps.create({
7  url: 'https://docs.olostep.com',
8  includeUrls: ['/features/**'],
9  topN: 100,
10})
11
12console.log(`Map ID: ${map.id}`)
13
14// Iterate all URLs (handles pagination automatically)
15for await (const url of map.urls()) {
16  console.log(url)
17}
1curl -s -X POST "https://api.olostep.com/v1/maps" \
2  -H "Authorization: Bearer <YOUR_API_KEY>" \
3  -H "Content-Type: application/json" \
4  -d '{
5    "url": "https://docs.olostep.com",
6    "include_urls": ["/features/**"],
7    "top_n": 100
8  }'

What you can build

Power enrichment, monitoring, and AI workflows from live web pages.

  • SEO audits

    Inventory indexable URLs and compare against sitemaps.

  • Site migrations

    Export URL lists for redirects and QA.

  • AI agents

    Give agents a bounded list of pages to read or summarize.

Explore more Use Cases

Trusted by Amazing Teams Building the Future of AI

"Olostep is the best API to search, extract, and structure data from the Web. Happy to be customers"

Trevor West
Co-founder & CEO Podqi

"Olostep has become the default Web Layer infrastructure for our company"

Richard He
Co-founder & CEO Openmart

"Olostep lets us turn any website into an API. Great product, great people"

Rob Hayes
Co-founder Merchkit

“One of the best teams in the space, with exceptional talent density. We love working with Olostep”

Alex Halliday
CEO AirOps
Know more

Start mapping the web

Get clean data for your AI from any website with Olostep
Most cost-effective API. Built for scale

Are you an AI Agent? Get started here

Frequently asked questions

Product & Capabilities
  • What is the Crawl endpoint for?

    It walks a site (within your limits) and returns page records you can later retrieve with /v1/retrieve — ideal for multi-page extraction without orchestrating every URL yourself.

  • How do I get page content?

    List pages from /v1/crawls/{id}/pages, then call /v1/retrieve with each page's retrieve_id. Prefer retrieve over deprecated inline content fields on pages.

Usage & Automation
  • Can I filter which URLs are crawled?

    Yes. Use include_urls and exclude_urls globs, search_query, top_n, and related parameters as documented.

Pricing & Plans
  • How much does crawling cost?

    Crawl is billed per page crawled (see current pricing in the docs).