Recursive crawling, structured

Web Crawling API for turning sites into datasets

Turn a single URL into a dataset of clean pages. Our recursive crawler handles robots.txt, JavaScript rendering, and page limits — perfect for RAG, AI training, and site migration.

No credit card500 free requestsNo-code friendly

Web Crawling API playground

Crawl a site. Inspect structured JSON.

Live crawl

https://docs.olostep.com

Submit a URL to start a live crawl. A crawl job runs asynchronously — this playground polls for status and pages as they're processed.

JSON structured data

{
  "start_url": "https://docs.olostep.com",
  "crawl_id": null,
  "status": null,
  "pages_count": 0,
  "pages": []
}

Developer experience

Create a job. Poll until it's done.

Crawls run asynchronously — start one with a URL and a page limit, then poll for status until it's complete and list the pages it found.

REST APIAPI key authenticationAsync job + pollingrobots.txt respectedJS renderingWebhook notifications

Read the docs

# 1. Start the crawl
curl --request POST \
  --url https://api.olostep.com/v1/crawls \
  --header 'Authorization: Bearer YOUR_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "start_url": "https://docs.olostep.com",
    "max_pages": 5,
    "include_urls": ["/**"]
  }'

# 2. Poll for status (repeat until status is "completed")
curl --url https://api.olostep.com/v1/crawls/CRAWL_ID \
  --header 'Authorization: Bearer YOUR_API_KEY'

# 3. List crawled pages
curl --url 'https://api.olostep.com/v1/crawls/CRAWL_ID/pages?cursor=0&limit=25' \
  --header 'Authorization: Bearer YOUR_API_KEY'

Trusted by the best startups in the world

The problem

Turning a website into usable data is harder than it looks

You don't know every URL

Sites don't ship a full list of pages — discovering them means following links yourself.

Modern sites need a browser

Single-page apps render content with JavaScript, so a plain HTTP fetch misses most of the page.

Scale needs queue management

Crawling hundreds of pages means concurrency, retries, and rate limits — easy to get wrong.

Async job, structured pages

Start a crawl, poll for results

Send a start URL and a page limit, poll the job until it's complete, and get back a clean list of every page it found.

Recursive discoveryPage & depth limitsURL include/exclude globsWebhook on completionrobots.txt respectedFull page content on request

{
  "id": "crawl_20tyuvpx78",
  "status": "completed",
  "pages_count": 3,
  "pages": [
    { "url": "https://example.com/", "retrieve_id": "ftfky1o5lo" },
    { "url": "https://example.com/docs", "retrieve_id": "gr427f2yny" }
  ]
}

Capabilities

Built to ingest whole sites, not just single pages

Recursive crawling

Start at one URL and follow links across the domain, up to a page or depth limit you control.

JS rendering

Pages are rendered with a real browser, so single-page apps and JS-heavy sites are captured fully.

robots.txt respected

Crawls follow robots.txt by default, so you stay within a site's stated crawling rules.

Structured page data

Each crawled page comes back with its URL and a retrieve_id for fetching full content.

Async job model

Start a crawl and poll for status — no long-held connection, no timeout on your side.

Batch-ready workflows

Crawl many domains on a schedule to keep datasets and knowledge bases current.

Use cases

Turn any site into a clean, usable dataset

RAG knowledge bases

Ingest documentation and help-center sites to ground AI models with real content.

SEO audits

Crawl a domain to review structure, content, and page coverage.

Content migration

Pull an entire site's content as clean pages ahead of a CMS or platform migration.

Market intelligence

Track competitor sites for new pages, pricing, or product changes.

Site mapping

Understand a domain's structure and internal linking before deeper analysis.

Offline datasets

Build a snapshot of a site's content for training or analysis.

How it works

From one URL to a full dataset

Start a crawl

Send a start URL, a page limit, and optional include/exclude patterns. Get a job id back immediately.

Recursive spidering

Olostep visits pages, follows links, and processes the domain up to your limits.

Retrieve pages

Poll for status, then list the pages found — each with a retrieve_id for full content.

Comparison

Build a crawler internally or use an API?

Requirement

Build internally

Use the API

Link discovery

Write a spider

Set a start URL

JS rendering

Run headless browsers yourself

Handled automatically

Rate limiting & retries

Build queue infrastructure

Managed for you

robots.txt handling

Implement and maintain

Respected by default

Page limits & filters

Custom logic

max_pages, include/exclude

Developer time

High ongoing cost

One request + polling

Ground your AI on an entire site, not one page

Ingest documentation, help centers, and knowledge bases in one crawl — clean pages ready for RAG pipelines, training sets, or migration.

Pricing

Web Crawling API pricing

Start with 500 free requests, then scale crawls as volume grows.

Starter

$9/ month

For testing, prototypes, and small crawls.

Everything in Free
5,000 successful requests
150 concurrent requests

Standard

$99/ month

For recurring crawls and knowledge bases.

Everything in Starter
200K successful requests
500 concurrent requests

Scale

$399/ month

For high-volume platforms and enterprise workflows.

Everything in Standard
1 million successful requests
AI-powered browser automations

Frequently asked questions

Product & Capabilities

What is a Web Crawling API?

A Web Crawling API starts at a URL and recursively follows links across a site, returning the content of every page it visits — up to a page count or depth limit you set.

What's the difference between crawling and scraping?

Scraping extracts data from a single, known URL. Crawling starts at one URL and discovers other pages on the site automatically, so you don't need the full list of URLs up front.

Does it handle JavaScript-rendered sites?

Yes. Pages are rendered with a real browser before content is captured, so single-page apps built with React, Next.js, and similar frameworks are handled correctly.

Usage & Automation

Does the crawler respect robots.txt?

Yes, by default. This can be overridden if you have permission to crawl a site more broadly.

How do I check when a crawl finishes?

Poll the crawl's status endpoint until status is completed, or provide a webhook_url to be notified automatically.

Can I limit or filter what gets crawled?

Yes. Set max_pages and max_depth to bound the crawl, and use include_urls / exclude_urls glob patterns to target specific sections of a site.

How do I get the content of a crawled page?

Each page in the crawl's results includes a retrieve_id — use it with the retrieve endpoint to fetch that page's full HTML or Markdown content.

Pricing & Plans

How is a crawl billed?

Billing is based on successfully processed pages — a 50-page crawl counts as 50 requests against your plan.

Do failed pages count against my limit?

No, only successfully processed pages are billed.

Is there a free tier?

Yes, 500 free requests on signup — enough to crawl several small sites or test the API.

Collect a site's content without maintaining crawling infrastructure

Start with one URL and connect clean, structured pages to RAG, training, migration, or research workflows.