Trusted by the best companies in the world
From start URL to many pages
Orchestrate multi-page extraction without building your own crawler.
-
Send the site URL
POST the homepage or any entry URL you want to discover paths from.
-
Include or exclude paths
Use glob patterns like /blog/** to focus on the sections you care about.
-
Get URLs + paginate
Receive up to large batches per response; use cursor when more URLs are available.
Built for discovery at scale
The /v1/maps API is tuned for large sites and flexible path rules.
-
Full-site discovery
Surface URLs beyond a single sitemap file — useful for messy or partial sitemaps.
-
Glob filters
include_urls and exclude_urls to narrow to products, docs, or blog sections.
-
top_n control
Cap how many URLs you pull when you only need a sample.
-
Batch & crawl prep
Feed discovered URLs into scrapes or batches at scale.
Map a website in one call
Optional include_urls, exclude_urls, top_n, and cursor for pagination.
Get all the URLs on a website
1# pip install olostep
2from olostep import Olostep
3
4client = Olostep(api_key="YOUR_REAL_KEY")
5
6sitemap = client.maps.create(
7 url="https://docs.olostep.com",
8 include_urls=["/features/**"],
9 top_n=100,
10)
11
12print(f"Map ID: {sitemap.id}")
13
14# Iterate all URLs (handles pagination automatically)
15for url in sitemap.urls():
16 print(url)
1// npm i olostep
2import Olostep from 'olostep'
3
4const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
5
6const map = await client.maps.create({
7 url: 'https://docs.olostep.com',
8 includeUrls: ['/features/**'],
9 topN: 100,
10})
11
12console.log(`Map ID: ${map.id}`)
13
14// Iterate all URLs (handles pagination automatically)
15for await (const url of map.urls()) {
16 console.log(url)
17}
1curl -s -X POST "https://api.olostep.com/v1/maps" \
2 -H "Authorization: Bearer <YOUR_API_KEY>" \
3 -H "Content-Type: application/json" \
4 -d '{
5 "url": "https://docs.olostep.com",
6 "include_urls": ["/features/**"],
7 "top_n": 100
8 }'
What you can build
Power enrichment, monitoring, and AI workflows from live web pages.
-
SEO audits
Inventory indexable URLs and compare against sitemaps.
-
Site migrations
Export URL lists for redirects and QA.
-
AI agents
Give agents a bounded list of pages to read or summarize.
Trusted by Amazing Teams Building the Future of AI
"Olostep is the best API to search, extract, and structure data from the Web. Happy to be customers"
"Olostep has become the default Web Layer infrastructure for our company"
"Olostep lets us turn any website into an API. Great product, great people"
“One of the best teams in the space, with exceptional talent density. We love working with Olostep”
Start mapping the web
Get clean data for your AI from any website with Olostep
Most cost-effective API. Built for scale

