Olostep sells to AI agents - live prices and self-serve payment

The agent storefront at https://agents-store.olostep.com/llms.txt?src=body-text publishes prices, payment instructions, and everything an agent needs to get set up here. Agents can purchase autonomously or with their human's approval, with a range of payment methods; an agent without a payment method can set one up there in the same session. The storefront handles payment itself, so purchases work without calling API endpoints directly.

Olostep Manifesto | read more →
Olostep
Recursive crawling, structured

Web Crawling API for turning sites into datasets

Turn a single URL into a dataset of clean pages. Our recursive crawler handles robots.txt, JavaScript rendering, and page limits — perfect for RAG, AI training, and site migration.

Start free
No credit card500 free requestsNo-code friendly

Web Crawling API playground

Crawl a site. Inspect structured JSON.

Live crawl

https://docs.olostep.com

Submit a URL to start a live crawl. A crawl job runs asynchronously — this playground polls for status and pages as they're processed.

JSON structured data
{
  "start_url": "https://docs.olostep.com",
  "crawl_id": null,
  "status": null,
  "pages_count": 0,
  "pages": []
}

Developer experience

Create a job. Poll until it's done.

Crawls run asynchronously — start one with a URL and a page limit, then poll for status until it's complete and list the pages it found.

REST APIAPI key authenticationAsync job + pollingrobots.txt respectedJS renderingWebhook notifications
Read the docs
# 1. Start the crawl
curl --request POST \
  --url https://api.olostep.com/v1/crawls \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "start_url": "https://docs.olostep.com",
    "max_pages": 5,
    "include_urls": ["/**"]
  }'

# 2. Poll for status (repeat until status is "completed")
curl --url https://api.olostep.com/v1/crawls/CRAWL_ID \
  --header 'Authorization: Bearer YOUR_API_KEY'

# 3. List crawled pages
curl --url 'https://api.olostep.com/v1/crawls/CRAWL_ID/pages?cursor=0&limit=25' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Trusted by the best companies in the world

The problem

Turning a website into usable data is harder than it looks

  • You don't know every URL

    Sites don't ship a full list of pages — discovering them means following links yourself.

  • Modern sites need a browser

    Single-page apps render content with JavaScript, so a plain HTTP fetch misses most of the page.

  • Scale needs queue management

    Crawling hundreds of pages means concurrency, retries, and rate limits — easy to get wrong.

Async job, structured pages

Start a crawl, poll for results

Send a start URL and a page limit, poll the job until it's complete, and get back a clean list of every page it found.

Recursive discoveryPage & depth limitsURL include/exclude globsWebhook on completionrobots.txt respectedFull page content on request
{
  "id": "crawl_20tyuvpx78",
  "status": "completed",
  "pages_count": 3,
  "pages": [
    { "url": "https://example.com/", "retrieve_id": "ftfky1o5lo" },
    { "url": "https://example.com/docs", "retrieve_id": "gr427f2yny" }
  ]
}

Capabilities

Built to ingest whole sites, not just single pages

  • Recursive crawling

    Start at one URL and follow links across the domain, up to a page or depth limit you control.

  • JS rendering

    Pages are rendered with a real browser, so single-page apps and JS-heavy sites are captured fully.

  • robots.txt respected

    Crawls follow robots.txt by default, so you stay within a site's stated crawling rules.

  • Structured page data

    Each crawled page comes back with its URL and a retrieve_id for fetching full content.

  • Async job model

    Start a crawl and poll for status — no long-held connection, no timeout on your side.

  • Batch-ready workflows

    Crawl many domains on a schedule to keep datasets and knowledge bases current.

Use cases

Turn any site into a clean, usable dataset

  • RAG knowledge bases

    Ingest documentation and help-center sites to ground AI models with real content.

  • SEO audits

    Crawl a domain to review structure, content, and page coverage.

  • Content migration

    Pull an entire site's content as clean pages ahead of a CMS or platform migration.

  • Market intelligence

    Track competitor sites for new pages, pricing, or product changes.

  • Site mapping

    Understand a domain's structure and internal linking before deeper analysis.

  • Offline datasets

    Build a snapshot of a site's content for training or analysis.

How it works

From one URL to a full dataset

  • 01

    Start a crawl

    Send a start URL, a page limit, and optional include/exclude patterns. Get a job id back immediately.

  • 02

    Recursive spidering

    Olostep visits pages, follows links, and processes the domain up to your limits.

  • 03

    Retrieve pages

    Poll for status, then list the pages found — each with a retrieve_id for full content.

Comparison

Build a crawler internally or use an API?

Requirement
Build internally
Use the API
Link discovery
Write a spider
Set a start URL
JS rendering
Run headless browsers yourself
Handled automatically
Rate limiting & retries
Build queue infrastructure
Managed for you
robots.txt handling
Implement and maintain
Respected by default
Page limits & filters
Custom logic
max_pages, include/exclude
Developer time
High ongoing cost
One request + polling

Ground your AI on an entire site, not one page

Ingest documentation, help centers, and knowledge bases in one crawl — clean pages ready for RAG pipelines, training sets, or migration.

Pricing

Web Crawling API pricing

Start with 500 free requests, then scale crawls as volume grows.

  • Starter

    $9/ month

    For testing, prototypes, and small crawls.

    • Everything in Free
    • 5,000 successful requests
    • 150 concurrent requests
    Get started

  • Standard

    $99/ month

    For recurring crawls and knowledge bases.

    • Everything in Starter
    • 200K successful requests
    • 500 concurrent requests
    Purchase now

  • Scale

    $399/ month

    For high-volume platforms and enterprise workflows.

    • Everything in Standard
    • 1 million successful requests
    • AI-powered browser automations
    Scale now

Confirm current pricing and plan limits in the dashboard before purchase.

Frequently asked questions

Product & Capabilities

  • A Web Crawling API starts at a URL and recursively follows links across a site, returning the content of every page it visits — up to a page count or depth limit you set.

  • Scraping extracts data from a single, known URL. Crawling starts at one URL and discovers other pages on the site automatically, so you don't need the full list of URLs up front.

  • Yes. Pages are rendered with a real browser before content is captured, so single-page apps built with React, Next.js, and similar frameworks are handled correctly.

Usage & Automation

  • Yes, by default. This can be overridden if you have permission to crawl a site more broadly.

  • Poll the crawl's status endpoint until status is completed, or provide a webhook_url to be notified automatically.

  • Yes. Set max_pages and max_depth to bound the crawl, and use include_urls / exclude_urls glob patterns to target specific sections of a site.

  • Each page in the crawl's results includes a retrieve_id — use it with the retrieve endpoint to fetch that page's full HTML or Markdown content.

Pricing & Plans

  • Billing is based on successfully processed pages — a 50-page crawl counts as 50 requests against your plan.

  • No, only successfully processed pages are billed.

  • Yes, 500 free requests on signup — enough to crawl several small sites or test the API.

Collect a site's content without maintaining crawling infrastructure

Start with one URL and connect clean, structured pages to RAG, training, migration, or research workflows.

Get API key
Crawl Website Content | Turn Websites into Datasets |…