Trusted by the best startups startups in the world
From start URL to many pages
Orchestrate multi-page extraction without building your own crawler.
Send the site URL
POST the homepage or any entry URL you want to discover paths from.
Include or exclude paths
Use glob patterns like /blog/** to focus on the sections you care about.
Get URLs + paginate
Receive up to large batches per response; use cursor when more URLs are available.
Built for discovery at scale
The /v1/maps API is tuned for large sites and flexible path rules.
Full-site discovery
Surface URLs beyond a single sitemap file — useful for messy or partial sitemaps.
Glob filters
include_urls and exclude_urls to narrow to products, docs, or blog sections.
top_n control
Cap how many URLs you pull when you only need a sample.
Batch & crawl prep
Feed discovered URLs into scrapes or batches at scale.
Map a website in one call
Optional include_urls, exclude_urls, top_n, and cursor for pagination.
Get all the URLs on a website
1# pip install olostep
2from olostep import Olostep
3
4client = Olostep(api_key="YOUR_REAL_KEY")
5
6sitemap = client.maps.create(
7 url="https://docs.olostep.com",
8 include_urls=["/features/**"],
9 top_n=100,
10)
11
12print(f"Map ID: {sitemap.id}")
13
14# Iterate all URLs (handles pagination automatically)
15for url in sitemap.urls():
16 print(url)1// npm i olostep
2import Olostep from 'olostep'
3
4const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
5
6const map = await client.maps.create({
7 url: 'https://docs.olostep.com',
8 includeUrls: ['/features/**'],
9 topN: 100,
10})
11
12console.log(`Map ID: ${map.id}`)
13
14// Iterate all URLs (handles pagination automatically)
15for await (const url of map.urls()) {
16 console.log(url)
17}1curl -s -X POST "https://api.olostep.com/v1/maps" \
2 -H "Authorization: Bearer <YOUR_API_KEY>" \
3 -H "Content-Type: application/json" \
4 -d '{
5 "url": "https://docs.olostep.com",
6 "include_urls": ["/features/**"],
7 "top_n": 100
8 }'
What you can build
Power enrichment, monitoring, and AI workflows from live web pages.
SEO audits
Inventory indexable URLs and compare against sitemaps.
Site migrations
Export URL lists for redirects and QA.
AI agents
Give agents a bounded list of pages to read or summarize.
Trusted by Amazing Teams Building the Future of AI
Olostep is the best!!! We automated entire data pipelines with just a prompt
Olostep has become the default Web Layer infrastructure for our company
Olostep works like a charm! And your customer service is exceptional
Olostep lets us turn any website into an API. Great product, great people
I highly recommend Olostep, great product!
We verify coupon codes at scale. Love Olostep. It works on any e-commerce
Olostep is the best API to search, extract, and structure data from the Web. Happy to be customers
We use /batches combined with parsers and it's magical how we can get structured data at large scale
Olostep allowed us to search and structure events data across the Web
Reliable and cost-effective API for working with data. Congrats on the cool product
Start mapping the web
Get clean data for your AI from any website with Olostep
Most cost-effective API. Built for scale
Frequently asked questions
What is the Crawl endpoint for?
It walks a site (within your limits) and returns page records you can later retrieve with /v1/retrieve — ideal for multi-page extraction without orchestrating every URL yourself.
How do I get page content?
List pages from /v1/crawls/{id}/pages, then call /v1/retrieve with each page's retrieve_id. Prefer retrieve over deprecated inline content fields on pages.
Can I filter which URLs are crawled?
Yes. Use include_urls and exclude_urls globs, search_query, top_n, and related parameters as documented.
How much does crawling cost?
Crawl is billed per page crawled (see current pricing in the docs).
