Trusted by the best startups startups in the world
From start URL to many pages
Orchestrate multi-page extraction without building your own crawler.
-
Send the site URL
POST the homepage or any entry URL you want to discover paths from.
-
Include or exclude paths
Use glob patterns like /blog/** to focus on the sections you care about.
-
Get URLs + paginate
Receive up to large batches per response; use cursor when more URLs are available.
Built for discovery at scale
The /v1/maps API is tuned for large sites and flexible path rules.
-
Full-site discovery
Surface URLs beyond a single sitemap file — useful for messy or partial sitemaps.
-
Glob filters
include_urls and exclude_urls to narrow to products, docs, or blog sections.
-
top_n control
Cap how many URLs you pull when you only need a sample.
-
Batch & crawl prep
Feed discovered URLs into scrapes or batches at scale.
Map a website in one call
Optional include_urls, exclude_urls, top_n, and cursor for pagination.
Get all the URLs on a website
1# pip install olostep
2from olostep import Olostep
3
4client = Olostep(api_key="YOUR_REAL_KEY")
5
6sitemap = client.maps.create(
7 url="https://docs.olostep.com",
8 include_urls=["/features/**"],
9 top_n=100,
10)
11
12print(f"Map ID: {sitemap.id}")
13
14# Iterate all URLs (handles pagination automatically)
15for url in sitemap.urls():
16 print(url)1// npm i olostep
2import Olostep from 'olostep'
3
4const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
5
6const map = await client.maps.create({
7 url: 'https://docs.olostep.com',
8 includeUrls: ['/features/**'],
9 topN: 100,
10})
11
12console.log(`Map ID: ${map.id}`)
13
14// Iterate all URLs (handles pagination automatically)
15for await (const url of map.urls()) {
16 console.log(url)
17}1curl -s -X POST "https://api.olostep.com/v1/maps" \
2 -H "Authorization: Bearer <YOUR_API_KEY>" \
3 -H "Content-Type: application/json" \
4 -d '{
5 "url": "https://docs.olostep.com",
6 "include_urls": ["/features/**"],
7 "top_n": 100
8 }'
What you can build
Power enrichment, monitoring, and AI workflows from live web pages.
-
SEO audits
Inventory indexable URLs and compare against sitemaps.
-
Site migrations
Export URL lists for redirects and QA.
-
AI agents
Give agents a bounded list of pages to read or summarize.
Trusted by Amazing Teams Building the Future of AI
-
Michelle JuliaCo-founder & CEO AuriumOlostep is the best!!! We automated entire data pipelines with just a prompt
-
Richard HeCo-founder & CEO OpenmartOlostep has become the default Web Layer infrastructure for our company
-
Max Brodeur-UrbasCo-founder & CEO GumloopOlostep works like a charm! And your customer service is exceptional
-
Rob HayesCo-founder MerchkitOlostep lets us turn any website into an API. Great product, great people
-
Brandon CohenCo-founder & CTO CivilGridI highly recommend Olostep, great product!
-
Co-founder & CEO Gedd.itWe verify coupon codes at scale. Love Olostep. It works on any e-commerce
-
Trevor WestCo-founder & CEO PodqiOlostep is the best API to search, extract, and structure data from the Web. Happy to be customers
-
Rida NaveedCo-founder ZecentoWe use /batches combined with parsers and it's magical how we can get structured data at large scale
-
Kieran V.Growth PlotsEventsOlostep allowed us to search and structure events data across the Web
-
Paul MitFounder FoundbaseReliable and cost-effective API for working with data. Congrats on the cool product
Start mapping the web
Get clean data for your AI from any website with Olostep
Most cost-effective API. Built for scale
Frequently asked questions
-
What is the Crawl endpoint for?
It walks a site (within your limits) and returns page records you can later retrieve with /v1/retrieve — ideal for multi-page extraction without orchestrating every URL yourself.
-
How do I get page content?
List pages from /v1/crawls/{id}/pages, then call /v1/retrieve with each page's retrieve_id. Prefer retrieve over deprecated inline content fields on pages.
-
Can I filter which URLs are crawled?
Yes. Use include_urls and exclude_urls globs, search_query, top_n, and related parameters as documented.
-
How much does crawling cost?
Crawl is billed per page crawled (see current pricing in the docs).