Trusted by the best companies in the world
From start URL to many pages
Orchestrate multi-page extraction without building your own crawler.
-
Start a crawl
POST start_url with include_urls, exclude_urls, max_pages, and optional max_depth.
-
Poll or webhook
Poll GET /v1/crawls/{id} until status is completed, or pass webhook_url for a push notification.
-
List pages & retrieve
Paginate GET /v1/crawls/{id}/pages, then fetch each retrieve_id via /v1/retrieve for markdown or HTML.
Built for multi-page jobs
The /v1/crawls API fits between one-off scrapes and huge batch URL lists.
-
Bounded exploration
Control depth and max_pages so crawls stay predictable.
-
Webhooks
Optional webhook_url when the crawl completes.
-
Cursor pagination
Stream pages while the crawl is in progress or after completion.
-
Retrieve pipeline
Pair crawls with /v1/retrieve for markdown, html, or json formats.
Start a crawl. Poll. List pages.
Use webhooks for completion signals; paginate pages with cursor and limit.
Crawl all the subpages
1# pip install olostep
2from olostep import Olostep
3
4client = Olostep(api_key="YOUR_REAL_KEY")
5
6crawl = client.crawls.create(
7 start_url="https://olostep.com",
8 max_pages=100,
9 include_urls=["/**"],
10 exclude_urls=["/collections/**"],
11 include_external=False,
12)
13
14print(crawl.id, crawl.status)
15
16# Wait for completion and iterate pages
17for page in crawl.pages():
18 print(page.url)
19 content = page.retrieve(["markdown"])
20 print(content.markdown_content[:200])
1// npm i olostep
2import Olostep from 'olostep'
3
4const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
5
6const crawl = await client.crawls.create({
7 url: 'https://olostep.com',
8 maxPages: 100,
9 includeUrls: ['/**'],
10 excludeUrls: ['/collections/**'],
11 includeExternal: false,
12})
13
14console.log(crawl.id, crawl.status)
15
16// Wait for completion and iterate pages
17for await (const page of crawl.pages()) {
18 console.log(page.url)
19 const content = await client.retrieve({ retrieveId: page.retrieve_id, formats: ['markdown'] })
20 console.log(content.markdown_content.slice(0, 200))
21}
1# Start crawl
2curl -s -X POST "https://api.olostep.com/v1/crawls" \
3 -H "Authorization: Bearer <YOUR_API_KEY>" \
4 -H "Content-Type: application/json" \
5 -d '{
6 "start_url": "https://olostep.com",
7 "max_pages": 100,
8 "include_urls": ["/**"],
9 "exclude_urls": ["/collections/**"],
10 "include_external": false
11 }'
12
13# Check status (replace <CRAWL_ID>)
14curl -s "https://api.olostep.com/v1/crawls/<CRAWL_ID>" \
15 -H "Authorization: Bearer <YOUR_API_KEY>"
16
17# Get pages (replace <CRAWL_ID>)
18curl -s "https://api.olostep.com/v1/crawls/<CRAWL_ID>/pages" \
19 -H "Authorization: Bearer <YOUR_API_KEY>"
20
21# Retrieve content (replace <RETRIEVE_ID>)
22curl -s -G "https://api.olostep.com/v1/retrieve" \
23 -H "Authorization: Bearer <YOUR_API_KEY>" \
24 --data-urlencode "retrieve_id=<RETRIEVE_ID>" \
25 --data-urlencode "formats=markdown"
What you can build
Power enrichment, monitoring, and AI workflows from live web pages.
-
Documentation mirrors
Pull every docs page for offline search or RAG.
-
E-commerce catalogs
Crawl product sections with include patterns.
-
Compliance archives
Snapshot many pages under a domain with audit metadata.
Trusted by Amazing Teams Building the Future of AI
"Olostep is the best API to search, extract, and structure data from the Web. Happy to be customers"
"Olostep has become the default Web Layer infrastructure for our company"
"Olostep lets us turn any website into an API. Great product, great people"
“One of the best teams in the space, with exceptional talent density. We love working with Olostep”
Start crawling with Olostep
Get clean data for your AI from any website with Olostep
Most cost-effective API. Built for scale

