Trusted by the best startups startups in the world
From URL list to retrieved content
Orchestrate multi-page extraction without building your own crawler.
Submit items
POST an array of items with custom_id and url — optionally a batch-level parser for structured JSON.
Wait for completion
Poll GET /v1/batches/{id} until completed — or pass a webhook URL on create to get an HTTP POST when the batch finishes (with retries).
List & retrieve
GET /v1/batches/{id}/items with cursor; then /v1/retrieve per retrieve_id for markdown or json.
Built for large lists
The /v1/batches API complements scrapes for high-volume workflows.
High throughput
Process large URL lists in one job instead of thousands of separate requests.
Predictable runtime
Jobs complete in roughly 5–8 minutes regardless of batch size (within limits).
Cursor pagination
Walk items with cursor and limit while the batch runs or after it finishes.
Parser-ready
Attach parsers to return normalized JSON across all items.
Completion webhooks
Skip tight polling: pass webhook on create for a POST when the batch completes, with automatic retries on failed delivery.
Batch & item metadata
Attach Stripe-style key-value metadata on the whole batch or on each item for tracing, filtering, and joins in your stack.
Webhooks & metadata
First-class options on POST /v1/batches so production pipelines
don't rely on polling alone and can carry your own context through every job.
Webhooks
Pass webhook with a public HTTPS URL (not localhost). When the batch completes, Olostep sends an HTTP POST with a structured event payload. Failed deliveries are retried with backoff (multiple attempts over about 30 minutes). Respond with 2xx within 30 seconds and use the event id to handle duplicates safely.
Webhooks API referenceMetadata
Add metadata at batch level (same request as items) and/or on each item for row-level tags — ideal for project IDs, pipeline stage, or correlating results with your warehouse. Keys and values follow documented limits (Stripe-style). Metadata is returned on subsequent GET responses; you can also merge-update batch metadata via PATCH.
Metadata API referenceStart a batch. Poll or webhook. Retrieve items.
Parser for JSON, cursor on items, plus optional webhook and
batch/item metadata — see Create batch in the API reference.
Process up to 10k URLs in one batch. Get results in 5-8 mins
1# pip install olostep
2from olostep import Olostep
3
4client = Olostep(api_key="YOUR_REAL_KEY")
5
6batch = client.batches.create(
7 urls=[
8 {"custom_id": "item-1", "url": "https://www.google.com/search?q=stripe&gl=us&hl=en"},
9 {"custom_id": "item-2", "url": "https://www.google.com/search?q=paddle&gl=us&hl=en"},
10 ],
11 parser="@olostep/google-search",
12)
13
14print(batch.id, batch.status)
15
16# Wait and iterate results (auto-waits for completion)
17for item in batch.items():
18 content = item.retrieve(["json"])
19 print(item.url, item.custom_id)
20 print(content.json_content)1// npm i olostep
2import Olostep from 'olostep'
3
4const client = new Olostep({ apiKey: 'YOUR_REAL_KEY' })
5
6const batch = await client.batches.create([
7 { url: 'https://www.google.com/search?q=stripe&gl=us&hl=en', customId: 'item-1' },
8 { url: 'https://www.google.com/search?q=paddle&gl=us&hl=en', customId: 'item-2' },
9], {
10 parser: '@olostep/google-search',
11})
12
13console.log(batch.id, batch.total_urls)
14
15// Wait and iterate results (auto-waits for completion)
16for await (const item of batch.items()) {
17 const content = await item.retrieve(['json'])
18 console.log(item.url, item.custom_id)
19 console.log(content.json_content)
20}1curl -s -X POST "https://api.olostep.com/v1/batches" \
2 -H "Authorization: Bearer <YOUR_API_KEY>" \
3 -H "Content-Type: application/json" \
4 -d '{
5 "items": [
6 {"custom_id": "item-1", "url": "https://www.google.com/search?q=stripe&gl=us&hl=en"},
7 {"custom_id": "item-2", "url": "https://www.google.com/search?q=paddle&gl=us&hl=en"}
8 ],
9 "parser": {"id": "@olostep/google-search"}
10 }'
What you can build
Monitoring, enrichment, and training-data collection at scale.
SERP & monitoring
Run many search or listing URLs through a parser in one batch.
Catalog ingestion
Feed product or directory URLs at scale.
Data pipelines
Hand off nightly jobs from your warehouse or orchestrator.
Trusted by Amazing Teams Building the Future of AI
Olostep is the best!!! We automated entire data pipelines with just a prompt
Olostep has become the default Web Layer infrastructure for our company
Olostep works like a charm! And your customer service is exceptional
Olostep lets us turn any website into an API. Great product, great people
I highly recommend Olostep, great product!
We verify coupon codes at scale. Love Olostep. It works on any e-commerce
Olostep is the best API to search, extract, and structure data from the Web. Happy to be customers
We use /batches combined with parsers and it's magical how we can get structured data at large scale
Olostep allowed us to search and structure events data across the Web
Reliable and cost-effective API for working with data. Congrats on the cool product
Start batching URLs
Get clean data for your AI from any website with Olostep
Most cost-effective API. Built for scale
Frequently asked questions
When should I use Batch vs parallel scrapes?
Batch is optimized for hundreds to tens of thousands of URLs with roughly constant wall-clock time. For small sets of URLs, parallel /v1/scrapes calls are often faster to complete.
Can I use Batch for e-commerce price tracking at huge scale?
Yes. Teams track millions of product pages by running many parallel batches (each within your account limits). Attach a batch-level parser so you only pull the JSON you need—price, reviews, seller, shipping time, availability—instead of full HTML, then join everything back using custom_id.
What about building an AI vertical search engine over millions of URLs and PDFs?
Batch fits large ingestion pipelines: submit huge URL lists (including pages that link to PDFs or docs), wait for the job to complete in predictable time, then list items with cursor and retrieve structured or markdown content per URL via /v1/retrieve. Scale out by splitting domains or cohorts across batches.
How does Batch help with SEO or GEO (tracking prompts on LLMs)?
Yes. Use include_urls and exclude_urls globs, search_query, top_n, and related parameters as documented.
What is custom_id?
Your stable identifier per row so you can join batch results back to your internal systems.
How do I get structured JSON?
Pass a parser on the batch (e.g. google-search parser) so items return JSON you can retrieve via /v1/retrieve.
How do batch webhooks work?
Include a public HTTPS webhook URL when you create the batch. Olostep POSTs a completion event (e.g. batch.completed) when processing finishes, with item counts and batch id in the payload. Failed deliveries retry with exponential backoff (up to 5 attempts over ~30 minutes). Return 2xx quickly and dedupe using the event id — see the webhooks docs for the envelope shape and best practices.
What is batch and item metadata for?
Metadata is optional key-value data (batch-level on the request body, or per item inside each object in items). Use it to tag jobs with project names, internal IDs, pipeline stage, or anything you need to correlate batches in your own systems. Values follow Stripe-style limits (e.g. max 50 keys). You can also merge-update metadata later via PATCH on the batch. See the metadata docs for validation rules.
Are there account limits?
New accounts may have a per-batch item limit; contact Olostep to raise it. See the docs warning for details.
