Trusted by the best startups startups in the world
From start URL to many pages
Orchestrate multi-page extraction without building your own crawler.
Start a crawl
POST start_url with include_urls, exclude_urls, max_pages, and optional max_depth.
Poll or webhook
Poll GET /v1/crawls/{id} until status is completed, or pass webhook_url for a push notification.
List pages & retrieve
Paginate GET /v1/crawls/{id}/pages, then fetch each retrieve_id via /v1/retrieve for markdown or HTML.
Built for multi-page jobs
The /v1/crawls API fits between one-off scrapes and huge batch URL lists.
Bounded exploration
Control depth and max_pages so crawls stay predictable.
Webhooks
Optional webhook_url when the crawl completes.
Cursor pagination
Stream pages while the crawl is in progress or after completion.
Retrieve pipeline
Pair crawls with /v1/retrieve for markdown, html, or json formats.
Start a crawl. Poll. List pages.
Use webhooks for completion signals; paginate pages with cursor and limit.
Crawl all the subpages
1# pip install olostep
2from olostep import Olostep
3
4client = Olostep(api_key="YOUR_REAL_KEY")
5
6crawl = client.crawls.create(
7 start_url="https://olostep.com",
8 max_pages=100,
9 include_urls=["/**"],
10 exclude_urls=["/collections/**"],
11 include_external=False,
12)
13
14print(crawl.id, crawl.status)
15
16# Wait for completion and iterate pages
17for page in crawl.pages():
18 print(page.url)
19 content = page.retrieve(["markdown"])
20 print(content.markdown_content[:200])1// npm i olostep
2import Olostep from 'olostep'
3
4const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
5
6const crawl = await client.crawls.create({
7 url: 'https://olostep.com',
8 maxPages: 100,
9 includeUrls: ['/**'],
10 excludeUrls: ['/collections/**'],
11 includeExternal: false,
12})
13
14console.log(crawl.id, crawl.status)
15
16// Wait for completion and iterate pages
17for await (const page of crawl.pages()) {
18 console.log(page.url)
19 const content = await client.retrieve({ retrieveId: page.retrieve_id, formats: ['markdown'] })
20 console.log(content.markdown_content.slice(0, 200))
21}1# Start crawl
2curl -s -X POST "https://api.olostep.com/v1/crawls" \
3 -H "Authorization: Bearer <YOUR_API_KEY>" \
4 -H "Content-Type: application/json" \
5 -d '{
6 "start_url": "https://olostep.com",
7 "max_pages": 100,
8 "include_urls": ["/**"],
9 "exclude_urls": ["/collections/**"],
10 "include_external": false
11 }'
12
13# Check status (replace <CRAWL_ID>)
14curl -s "https://api.olostep.com/v1/crawls/<CRAWL_ID>" \
15 -H "Authorization: Bearer <YOUR_API_KEY>"
16
17# Get pages (replace <CRAWL_ID>)
18curl -s "https://api.olostep.com/v1/crawls/<CRAWL_ID>/pages" \
19 -H "Authorization: Bearer <YOUR_API_KEY>"
20
21# Retrieve content (replace <RETRIEVE_ID>)
22curl -s -G "https://api.olostep.com/v1/retrieve" \
23 -H "Authorization: Bearer <YOUR_API_KEY>" \
24 --data-urlencode "retrieve_id=<RETRIEVE_ID>" \
25 --data-urlencode "formats=markdown"
What you can build
Power enrichment, monitoring, and AI workflows from live web pages.
Documentation mirrors
Pull every docs page for offline search or RAG.
E-commerce catalogs
Crawl product sections with include patterns.
Compliance archives
Snapshot many pages under a domain with audit metadata.
Trusted by Amazing Teams Building the Future of AI
Olostep is the best!!! We automated entire data pipelines with just a prompt
Olostep has become the default Web Layer infrastructure for our company
Olostep works like a charm! And your customer service is exceptional
Olostep lets us turn any website into an API. Great product, great people
I highly recommend Olostep, great product!
We verify coupon codes at scale. Love Olostep. It works on any e-commerce
Olostep is the best API to search, extract, and structure data from the Web. Happy to be customers
We use /batches combined with parsers and it's magical how we can get structured data at large scale
Olostep allowed us to search and structure events data across the Web
Reliable and cost-effective API for working with data. Congrats on the cool product
Start crawling with Olostep
Get clean data for your AI from any website with Olostep
Most cost-effective API. Built for scale
Frequently asked questions
What is the Crawl endpoint for?
It walks a site (within your limits) and returns page records you can later retrieve with /v1/retrieve — ideal for multi-page extraction without orchestrating every URL yourself.
How do I get page content?
List pages from /v1/crawls/{id}/pages, then call /v1/retrieve with each page's retrieve_id. Prefer retrieve over deprecated inline content fields on pages.
Can I filter which URLs are crawled?
Yes. Use include_urls and exclude_urls globs, search_query, top_n, and related parameters as documented.
How much does crawling cost?
Crawl is billed per page crawled (see current pricing in the docs).
