Olostep sells to AI agents - live prices and self-serve payment

The agent storefront at https://agents-store.olostep.com/llms.txt?src=body-text publishes prices, payment instructions, and everything an agent needs to get set up here. Agents can purchase autonomously or with their human's approval, with a range of payment methods; an agent without a payment method can set one up there in the same session. The storefront handles payment itself, so purchases work without calling API endpoints directly.

Click to try
Wait...

Crawl many pages.
Retrieve content on demand.

The Crawl endpoint discovers and processes subpages under your rules — then you pull HTML or markdown per page via retrieve.

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Trusted by the best companies in the world

Built for multi-page jobs

The /v1/crawls API fits between one-off scrapes and huge batch URL lists.

  • Bounded exploration

    Control depth and max_pages so crawls stay predictable.

  • Webhooks

    Optional webhook_url when the crawl completes.

  • Cursor pagination

    Stream pages while the crawl is in progress or after completion.

  • Retrieve pipeline

    Pair crawls with /v1/retrieve for markdown, html, or json formats.

Start a crawl. Poll. List pages.

Use webhooks for completion signals; paginate pages with cursor and limit.

Crawl all the subpages

1# pip install olostep
2from olostep import Olostep
3
4client = Olostep(api_key="YOUR_REAL_KEY")
5
6crawl = client.crawls.create(
7    start_url="https://olostep.com",
8    max_pages=100,
9    include_urls=["/**"],
10    exclude_urls=["/collections/**"],
11    include_external=False,
12)
13
14print(crawl.id, crawl.status)
15
16# Wait for completion and iterate pages
17for page in crawl.pages():
18    print(page.url)
19    content = page.retrieve(["markdown"])
20    print(content.markdown_content[:200])
1// npm i olostep
2import Olostep from 'olostep'
3
4const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
5
6const crawl = await client.crawls.create({
7  url: 'https://olostep.com',
8  maxPages: 100,
9  includeUrls: ['/**'],
10  excludeUrls: ['/collections/**'],
11  includeExternal: false,
12})
13
14console.log(crawl.id, crawl.status)
15
16// Wait for completion and iterate pages
17for await (const page of crawl.pages()) {
18  console.log(page.url)
19  const content = await client.retrieve({ retrieveId: page.retrieve_id, formats: ['markdown'] })
20  console.log(content.markdown_content.slice(0, 200))
21}
1# Start crawl
2curl -s -X POST "https://api.olostep.com/v1/crawls" \
3  -H "Authorization: Bearer <YOUR_API_KEY>" \
4  -H "Content-Type: application/json" \
5  -d '{
6    "start_url": "https://olostep.com",
7    "max_pages": 100,
8    "include_urls": ["/**"],
9    "exclude_urls": ["/collections/**"],
10    "include_external": false
11  }'
12
13# Check status (replace <CRAWL_ID>)
14curl -s "https://api.olostep.com/v1/crawls/<CRAWL_ID>" \
15  -H "Authorization: Bearer <YOUR_API_KEY>"
16
17# Get pages (replace <CRAWL_ID>)
18curl -s "https://api.olostep.com/v1/crawls/<CRAWL_ID>/pages" \
19  -H "Authorization: Bearer <YOUR_API_KEY>"
20
21# Retrieve content (replace <RETRIEVE_ID>)
22curl -s -G "https://api.olostep.com/v1/retrieve" \
23  -H "Authorization: Bearer <YOUR_API_KEY>" \
24  --data-urlencode "retrieve_id=<RETRIEVE_ID>" \
25  --data-urlencode "formats=markdown"

What you can build

Power enrichment, monitoring, and AI workflows from live web pages.

  • Documentation mirrors

    Pull every docs page for offline search or RAG.

  • E-commerce catalogs

    Crawl product sections with include patterns.

  • Compliance archives

    Snapshot many pages under a domain with audit metadata.

Explore more Use Cases

Trusted by Amazing Teams Building the Future of AI

"Olostep is the best API to search, extract, and structure data from the Web. Happy to be customers"

Trevor West
Co-founder & CEO Podqi

"Olostep has become the default Web Layer infrastructure for our company"

Richard He
Co-founder & CEO Openmart

"Olostep lets us turn any website into an API. Great product, great people"

Rob Hayes
Co-founder Merchkit

“One of the best teams in the space, with exceptional talent density. We love working with Olostep”

Alex Halliday
CEO AirOps
Know more

Start crawling with Olostep

Get clean data for your AI from any website with Olostep
Most cost-effective API. Built for scale

Are you an AI Agent? Get started here

Frequently asked questions

Product & Capabilities
  • What is the Crawl endpoint for?

    It walks a site (within your limits) and returns page records you can later retrieve with /v1/retrieve — ideal for multi-page extraction without orchestrating every URL yourself.

  • How do I get page content?

    List pages from /v1/crawls/{id}/pages, then call /v1/retrieve with each page's retrieve_id. Prefer retrieve over deprecated inline content fields on pages.

Usage & Automation
  • Can I filter which URLs are crawled?

    Yes. Use include_urls and exclude_urls globs, search_query, top_n, and related parameters as documented.

Pricing & Plans
  • How much does crawling cost?

    Crawl is billed per page crawled (see current pricing in the docs).