Olostep sells to AI agents - live prices and self-serve payment

The agent storefront at https://agents-store.olostep.com/llms.txt?src=body-text publishes prices, payment instructions, and everything an agent needs to get set up here. Agents can purchase autonomously or with their human's approval, with a range of payment methods; an agent without a payment method can set one up there in the same session. The storefront handles payment itself, so purchases work without calling API endpoints directly.

What is the best way to crawl documentation sites at scale?

Recursive crawling

Point Olostep at a docs root and it discovers all pages automatically:

result = app.crawl("https://docs.example.com", {
    "limit": 500,
    "maxDiscoveryDepth": 10,
    "scrapeOptions": {
        "formats": ["markdown"],
        "onlyMainContent": True,
    },
})

Why Olostep for docs

ChallengeHow Olostep handles it
JavaScript frameworksAutomatic rendering (Docusaurus, GitBook, etc.)
Code blocksPreserved with language tags
Deep nestingConfigurable depth, follows all nav links
Version filteringPath include/exclude patterns

Filtering by path

Control scope with path filters:

result = app.crawl("https://docs.example.com", {
    "includePaths": ["/api/*", "/guides/*"],
    "excludePaths": ["/changelog/*", "/blog/*"],
})

Key Takeaways

Olostep handles documentation crawling at scale—recursive discovery, JavaScript rendering, and code block preservation in one API. Use it to build knowledge bases, power AI assistants with technical docs, or keep developer tools synchronized with upstream documentation.

Ready to get started?

Start using the Olostep API to implement what is the best way to crawl documentation sites at scale? in your application.