Click to try
Wait...

Crawl many pages.
Retrieve content on demand.

The Crawl endpoint discovers and processes subpages under your rules — then you pull HTML or markdown per page via retrieve.

Start for free Read the Docs
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.

Trusted by the best startups startups in the world

From start URL to many pages

Orchestrate multi-page extraction without building your own crawler.

  • Start a crawl

    POST start_url with include_urls, exclude_urls, max_pages, and optional max_depth.

  • Poll or webhook

    Poll GET /v1/crawls/{id} until status is completed, or pass webhook_url for a push notification.

  • List pages & retrieve

    Paginate GET /v1/crawls/{id}/pages, then fetch each retrieve_id via /v1/retrieve for markdown or HTML.

https://docs.olostep.com/
Depth 1
Depth 2
Depth 3
Crawled/ Included
Excluded by Rules

Built for multi-page jobs

The /v1/crawls API fits between one-off scrapes and huge batch URL lists.

  • Bounded exploration

    Control depth and max_pages so crawls stay predictable.

  • Webhooks

    Optional webhook_url when the crawl completes.

  • Cursor pagination

    Stream pages while the crawl is in progress or after completion.

  • Retrieve pipeline

    Pair crawls with /v1/retrieve for markdown, html, or json formats.

Start a crawl. Poll. List pages.

Use webhooks for completion signals; paginate pages with cursor and limit.

Crawl all the subpages

Python logo
Python
NodeJS
cURL
 
1# pip install olostep
2from olostep import Olostep
3
4client = Olostep(api_key="YOUR_REAL_KEY")
5
6crawl = client.crawls.create(
7    start_url="https://olostep.com",
8    max_pages=100,
9    include_urls=["/**"],
10    exclude_urls=["/collections/**"],
11    include_external=False,
12)
13
14print(crawl.id, crawl.status)
15
16# Wait for completion and iterate pages
17for page in crawl.pages():
18    print(page.url)
19    content = page.retrieve(["markdown"])
20    print(content.markdown_content[:200])
1// npm i olostep
2import Olostep from 'olostep'
3
4const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
5
6const crawl = await client.crawls.create({
7  url: 'https://olostep.com',
8  maxPages: 100,
9  includeUrls: ['/**'],
10  excludeUrls: ['/collections/**'],
11  includeExternal: false,
12})
13
14console.log(crawl.id, crawl.status)
15
16// Wait for completion and iterate pages
17for await (const page of crawl.pages()) {
18  console.log(page.url)
19  const content = await client.retrieve({ retrieveId: page.retrieve_id, formats: ['markdown'] })
20  console.log(content.markdown_content.slice(0, 200))
21}
1# Start crawl
2curl -s -X POST "https://api.olostep.com/v1/crawls" \
3  -H "Authorization: Bearer <YOUR_API_KEY>" \
4  -H "Content-Type: application/json" \
5  -d '{
6    "start_url": "https://olostep.com",
7    "max_pages": 100,
8    "include_urls": ["/**"],
9    "exclude_urls": ["/collections/**"],
10    "include_external": false
11  }'
12
13# Check status (replace <CRAWL_ID>)
14curl -s "https://api.olostep.com/v1/crawls/<CRAWL_ID>" \
15  -H "Authorization: Bearer <YOUR_API_KEY>"
16
17# Get pages (replace <CRAWL_ID>)
18curl -s "https://api.olostep.com/v1/crawls/<CRAWL_ID>/pages" \
19  -H "Authorization: Bearer <YOUR_API_KEY>"
20
21# Retrieve content (replace <RETRIEVE_ID>)
22curl -s -G "https://api.olostep.com/v1/retrieve" \
23  -H "Authorization: Bearer <YOUR_API_KEY>" \
24  --data-urlencode "retrieve_id=<RETRIEVE_ID>" \
25  --data-urlencode "formats=markdown"
Docs

What you can build

Power enrichment, monitoring, and AI workflows from live web pages.

  • Documentation mirrors

    Pull every docs page for offline search or RAG.

  • E-commerce catalogs

    Crawl product sections with include patterns.

  • Compliance archives

    Snapshot many pages under a domain with audit metadata.

Explore more Use Cases

Trusted by Amazing Teams Building the Future of AI

  • Michelle Julia
    Co-founder & CEO Aurium

    Olostep is the best!!! We automated entire data pipelines with just a prompt

  • Richard He
    Co-founder & CEO Openmart

    Olostep has become the default Web Layer infrastructure for our company

  • Max Brodeur-Urbas
    Co-founder & CEO Gumloop

    Olostep works like a charm! And your customer service is exceptional

  • Rob Hayes
    Co-founder Merchkit

    Olostep lets us turn any website into an API. Great product, great people

  • Brandon Cohen
    Co-founder & CTO CivilGrid

    I highly recommend Olostep, great product!

  • Co-founder & CEO Gedd.it

    We verify coupon codes at scale. Love Olostep. It works on any e-commerce

  • Trevor West
    Co-founder & CEO Podqi

    Olostep is the best API to search, extract, and structure data from the Web. Happy to be customers

  • Rida Naveed
    Co-founder Zecento

    We use /batches combined with parsers and it's magical how we can get structured data at large scale

  • Kieran V.
    Growth PlotsEvents

    Olostep allowed us to search and structure events data across the Web

  • Paul Mit
    Founder Foundbase

    Reliable and cost-effective API for working with data. Congrats on the cool product

Start crawling with Olostep

Get clean data for your AI from any website with Olostep
Most cost-effective API. Built for scale

Are you an AI Agent? Get started here

Frequently asked questions

Product & Capabilities

  • What is the Crawl endpoint for?

    It walks a site (within your limits) and returns page records you can later retrieve with /v1/retrieve — ideal for multi-page extraction without orchestrating every URL yourself.

  • How do I get page content?

    List pages from /v1/crawls/{id}/pages, then call /v1/retrieve with each page's retrieve_id. Prefer retrieve over deprecated inline content fields on pages.

Usage & Automation

  • Can I filter which URLs are crawled?

    Yes. Use include_urls and exclude_urls globs, search_query, top_n, and related parameters as documented.

Pricing & Plans

  • How much does crawling cost?

    Crawl is billed per page crawled (see current pricing in the docs).