Web Crawling API for turning sites into datasets
Turn a single URL into a dataset of clean pages. Our recursive crawler handles robots.txt, JavaScript rendering, and page limits — perfect for RAG, AI training, and site migration.
Web Crawling API playground
Crawl a site. Inspect structured JSON.
https://docs.olostep.com
Submit a URL to start a live crawl. A crawl job runs asynchronously — this playground polls for status and pages as they're processed.
{
"start_url": "https://docs.olostep.com",
"crawl_id": null,
"status": null,
"pages_count": 0,
"pages": []
}Developer experience
Create a job. Poll until it's done.
Crawls run asynchronously — start one with a URL and a page limit, then poll for status until it's complete and list the pages it found.
Trusted by the best startups in the world
The problem
Turning a website into usable data is harder than it looks
You don't know every URL
Sites don't ship a full list of pages — discovering them means following links yourself.
Modern sites need a browser
Single-page apps render content with JavaScript, so a plain HTTP fetch misses most of the page.
Scale needs queue management
Crawling hundreds of pages means concurrency, retries, and rate limits — easy to get wrong.
Async job, structured pages
Start a crawl, poll for results
Send a start URL and a page limit, poll the job until it's complete, and get back a clean list of every page it found.
Capabilities
Built to ingest whole sites, not just single pages
Recursive crawling
Start at one URL and follow links across the domain, up to a page or depth limit you control.
JS rendering
Pages are rendered with a real browser, so single-page apps and JS-heavy sites are captured fully.
robots.txt respected
Crawls follow robots.txt by default, so you stay within a site's stated crawling rules.
Structured page data
Each crawled page comes back with its URL and a retrieve_id for fetching full content.
Async job model
Start a crawl and poll for status — no long-held connection, no timeout on your side.
Batch-ready workflows
Crawl many domains on a schedule to keep datasets and knowledge bases current.
Use cases
Turn any site into a clean, usable dataset
RAG knowledge bases
Ingest documentation and help-center sites to ground AI models with real content.
SEO audits
Crawl a domain to review structure, content, and page coverage.
Content migration
Pull an entire site's content as clean pages ahead of a CMS or platform migration.
Market intelligence
Track competitor sites for new pages, pricing, or product changes.
Site mapping
Understand a domain's structure and internal linking before deeper analysis.
Offline datasets
Build a snapshot of a site's content for training or analysis.
How it works
From one URL to a full dataset
- 01
Start a crawl
Send a start URL, a page limit, and optional include/exclude patterns. Get a job id back immediately.
- 02
Recursive spidering
Olostep visits pages, follows links, and processes the domain up to your limits.
- 03
Retrieve pages
Poll for status, then list the pages found — each with a retrieve_id for full content.
Comparison
Build a crawler internally or use an API?
Ground your AI on an entire site, not one page
Ingest documentation, help centers, and knowledge bases in one crawl — clean pages ready for RAG pipelines, training sets, or migration.
Pricing
Web Crawling API pricing
Start with 500 free requests, then scale crawls as volume grows.
Starter
$9/ monthFor testing, prototypes, and small crawls.
- Everything in Free
- 5,000 successful requests
- 150 concurrent requests
Standard
$99/ monthFor recurring crawls and knowledge bases.
- Everything in Starter
- 200K successful requests
- 500 concurrent requests
Scale
$399/ monthFor high-volume platforms and enterprise workflows.
- Everything in Standard
- 1 million successful requests
- AI-powered browser automations
Confirm current pricing and plan limits in the dashboard before purchase.
Frequently asked questions
Product & Capabilities
- What is a Web Crawling API?
A Web Crawling API starts at a URL and recursively follows links across a site, returning the content of every page it visits — up to a page count or depth limit you set.
- What's the difference between crawling and scraping?
Scraping extracts data from a single, known URL. Crawling starts at one URL and discovers other pages on the site automatically, so you don't need the full list of URLs up front.
- Does it handle JavaScript-rendered sites?
Yes. Pages are rendered with a real browser before content is captured, so single-page apps built with React, Next.js, and similar frameworks are handled correctly.
Usage & Automation
- Does the crawler respect robots.txt?
Yes, by default. This can be overridden if you have permission to crawl a site more broadly.
- How do I check when a crawl finishes?
Poll the crawl's status endpoint until status is completed, or provide a webhook_url to be notified automatically.
- Can I limit or filter what gets crawled?
Yes. Set max_pages and max_depth to bound the crawl, and use include_urls / exclude_urls glob patterns to target specific sections of a site.
- How do I get the content of a crawled page?
Each page in the crawl's results includes a retrieve_id — use it with the retrieve endpoint to fetch that page's full HTML or Markdown content.
Pricing & Plans
- How is a crawl billed?
Billing is based on successfully processed pages — a 50-page crawl counts as 50 requests against your plan.
- Do failed pages count against my limit?
No, only successfully processed pages are billed.
- Is there a free tier?
Yes, 500 free requests on signup — enough to crawl several small sites or test the API.
Collect a site's content without maintaining crawling infrastructure
Start with one URL and connect clean, structured pages to RAG, training, migration, or research workflows.