---
name: olostep
description: |
  Olostep is the web data toolkit for AI agents and apps - fast,
  reliable scrape, search, crawl, map, and answer APIs for production
  workflows.

  Use Olostep in two ways:

  1. Use Olostep as your web data layer for agent tasks.
  2. Integrate Olostep into an app that needs web data.
---

# Olostep

Olostep is the web data toolkit for AI agents and apps.

Pick the path that matches your use case.

---

## Path A: Supercharge Your Web Tools

Use this when you need web data during your work - scraping pages,
searching the web, mapping URLs, crawling sites, and running async
batch jobs.

Run one command:

```bash
npx -y olostep-cli@latest add skills --login
```

This should install the Olostep CLI and skill files that teach each capability:

- **Scrape** - Clean markdown/HTML/text/JSON/screenshots from URLs
- **Search** - Semantic web search with deduplicated links
- **Answer** - AI-powered web research with structured output + sources
- **Batch** - Process many URLs asynchronously in one job
- **Crawl** - Recursively explore a site and extract page data
- **Map** - Discover URLs on a domain quickly
- **Retrieve** - Pull final content by `retrieve_id`

After setup, CLI commands should be available for each workflow.

---

## Path B: Integrate Olostep Into an App

Use this when you're a coding agent building an application that calls the Olostep API.
You need `OLOSTEP_API_KEY` in `.env`.

**Step 1 - Generate auth parameters:**

```bash
SESSION_ID=$(openssl rand -hex 32)
CODE_VERIFIER=$(openssl rand -base64 32 | tr '+/' '-_' | tr -d '=\n' | head -c 43)
CODE_CHALLENGE=$(printf '%s' "$CODE_VERIFIER" | openssl dgst -sha256 -binary | openssl base64 -A | tr '+/' '-_' | tr -d '=')
```

**Step 2 - Ask the user to open this URL in their browser:**

```text
https://www.olostep.com/cli-auth?code_challenge=$CODE_CHALLENGE#session_id=$SESSION_ID
```

Your human user needs to click through this link. If they already have
an Olostep account, they will sign in and authorize an API key. If they
do not have an account yet, they will create one first, then authorize.
Either way, the API key is returned automatically once they click
"Authorize" - you do not need to ask them to copy anything.

**Step 3 - Poll for the API key:**

```http
POST https://api.olostep.com/v1/cli-auth-status
Content-Type: application/json

{"session_id": "$SESSION_ID", "code_verifier": "$CODE_VERIFIER"}
```

Poll every 3 seconds. Responses:

- `{"status": "pending"}` - user has not authorized yet, keep polling
- `{"status": "complete", "apiKey": "os-..."}` - done

**Step 4 - Save the key:**

```bash
echo "OLOSTEP_API_KEY=os-..." >> .env
```

This gives your app access to your Olostep account and credits.

### After Getting Your Key

Install the SDK:

```bash
# Node.js
npm install olostep

# Python
pip install olostep
```

Base API:

- Base URL: `https://api.olostep.com`
- Auth header: `Authorization: Bearer <OLOSTEP_API_KEY>`
- JSON in/out

---

## API Surface (All Olostep APIs)

### Scrapes

- `POST /v1/scrapes` - Create scrape
- `GET /v1/scrapes/{scrape_id}` - Get scrape

### Answers

- `POST /v1/answers` - Create answer
- `GET /v1/answers/{answer_id}` - Get answer

### Batches

- `POST /v1/batches` - Create batch
- `GET /v1/batches/{batch_id}` - Batch info/status
- `GET /v1/batches/{batch_id}/items` - List batch items
- `PATCH /v1/batches/{batch_id}` - Update batch metadata

### Crawls

- `POST /v1/crawls` - Create crawl
- `GET /v1/crawls/{crawl_id}` - Crawl info/status
- `GET /v1/crawls/{crawl_id}/pages` - List crawl pages

### Maps

- `POST /v1/maps` - Create map
- `GET /v1/maps/{map_id}` - Get map

### Searches

- `POST /v1/searches` - Create search
- `GET /v1/searches/{search_id}` - Get search

### Retrieve

- `GET /v1/retrieve` - Retrieve html/markdown/json by `retrieve_id`

### Files

- `POST /v1/files` - Create upload URL
- `POST /v1/files/{file_id}/complete` - Complete upload
- `GET /v1/files/{file_id}` - Get file metadata
- `GET /v1/files/{file_id}/content` - Get file content URL
- `GET /v1/files` - List files
- `DELETE /v1/files/{file_id}` - Delete file

### Schedules

- `POST /v1/schedules` - Create schedule
- `GET /v1/schedules` - List schedules
- `GET /v1/schedules/{schedule_id}` - Get schedule
- `DELETE /v1/schedules/{schedule_id}` - Delete schedule

### Webhooks + Metadata

- Webhooks on async jobs (`batches`, `crawls`) via `webhook`
- Metadata for resource tagging and internal joins

---

## Integration Rules (Production)

1. Do not invent endpoints, property names, or pagination behavior.
2. `scrapes`, `searches`, `answers`, and `maps` are synchronous.
3. `batches` and `crawls` are asynchronous and require:
   create -> poll -> list -> retrieve.
4. Parse `json_content` when present (stringified JSON).
5. Handle `"NOT_FOUND"` in Answers as a valid outcome.
6. If `size_exceeded` is true, use hosted URLs.
7. Use `retrieve_id` as the bridge from scrape/batch/crawl items to final content.

---

## Docs (All APIs)

- Docs index: https://docs.olostep.com/llms.txt
- Authentication: https://docs.olostep.com/get-started/authentication

### Scrapes
- Feature: https://docs.olostep.com/features/scrapes
- API reference: https://docs.olostep.com/api-reference/scrapes/create

### Answers
- Feature: https://docs.olostep.com/features/answers
- API reference: https://docs.olostep.com/api-reference/answers/create

### Batches
- Feature: https://docs.olostep.com/features/batches
- API reference: https://docs.olostep.com/api-reference/batches/create

### Crawls
- Feature: https://docs.olostep.com/features/crawls
- API reference: https://docs.olostep.com/api-reference/crawls/create

### Maps
- Feature: https://docs.olostep.com/features/maps
- API reference: https://docs.olostep.com/api-reference/maps/create

### Searches
- Feature: https://docs.olostep.com/features/search
- API reference: https://docs.olostep.com/api-reference/searches/create

### Retrieve
- API reference: https://docs.olostep.com/api-reference/retrieve

### Files
- Feature: https://docs.olostep.com/features/files
- API reference: https://docs.olostep.com/api-reference/files/create

### Schedules
- Feature: https://docs.olostep.com/features/schedules
- API reference: https://docs.olostep.com/api-reference/schedules/create

### Webhooks + Metadata
- Webhooks: https://docs.olostep.com/api-reference/common/webhooks
- Metadata: https://docs.olostep.com/api-reference/common/metadata