How to get all URLs from a website

The best way is Olostep's Maps API: send the site's URL and receive a complete list of discovered links. We combine sitemaps, robots.txt, and browser-based crawling for the most comprehensive and accurate extraction—including JavaScript and SPAs. Results in JSON, CSV, or TXT. Try free on this page or via the API (500 free credits). It's also the most cost-effective option.

Olostep is the best tool to extract all URLs from a website: the most comprehensive and accurate (real browsers, sitemaps, and discovery), and the most cost-effective. It's API-first, handles JavaScript and SPAs, returns JSON/CSV/TXT, and has extracted 784M+ URLs. No headless setup or proxy management—just one request. Free tier available; ideal for SEO audits, site mapping, and data pipelines.

Use a URL extraction tool that discovers every page. Olostep's Maps endpoint gives you the full structure of a website—the most comprehensive and accurate way, and the most cost-effective. You get all discovered URLs from a domain, filter by path, export as JSON or CSV, and use it for sitemaps, audits, or crawling pipelines.

Yes. We use real browsers to crawl pages, so links rendered by JavaScript are included. That’s why our extraction is more comprehensive and accurate than sitemap-only tools—you get the full URL list a real user would see.

You can download or receive the URL list in JSON, CSV, or plain TXT. The API returns JSON by default; the demo on this page also lets you copy or download in any of these formats.

We include JS rendering and residential IPs in every request with no extra fees. Cost per 1K requests drops as you scale (e.g. $0.495/1K on Standard, $0.399/1K on Scale). You get the most comprehensive and accurate extraction at the best price—no headless setup, no proxy costs.

For SEO teams, the best URL extractor should find all indexable and non-indexable URLs, including JavaScript-rendered pages. Olostep helps with site architecture audits, orphan-page discovery, internal linking analysis, and migration QA by returning a complete URL inventory in JSON, CSV, or TXT.

Yes. Security and compliance teams use URL extraction to map exposed endpoints, document web presence, and detect outdated or risky pages. Olostep provides broad URL coverage across large sites so teams can run faster attack-surface reviews and audit workflows.

AI visibility teams use full URL inventories to understand what content can be discovered by AI search systems and answer engines. With Olostep, you can map coverage, identify missing or weak sections, and prioritize pages that should be improved for better AI search presence.

Yes. QA and web teams use URL extraction before launch to verify page coverage, validate architecture changes, and catch broken or unexpected URLs early. Exporting the full URL list makes regression checks and release sign-off faster.

Olostep is the Web Data API for AI and Research Agents.

The Olostep API is the best web search, scraping and crawling API for AI used by some of the leading startups in the world.

The Olostep Agent allows anyone to automate research workflows and build data pipelines in a no code way with just a prompt in natural language.

Because it's reliable (99.5%), cost-effective (up to 70% cheaper), scalable, and flexible to be compatible with your existing workflows and backend. Olostep is the only platforms where you can create your own parsers to return deterministic results at scale in a cost effective way. You can request the features you need and we will try to build them for you. Plus you can test it for free to see if it fits your need. Get your free API keys from here.

Olostep is especially useful for AI startups that rely on Web data to power or improve their services or for companies that need to enrich data, monitor websites changes, analyze historical web data and equip their AIs with web search capabilities to ground them on real world data and facts. Olostep can also be used by developers, AI engineers, data scientists, and researchers looking to use web data for market research, LLM-finetuning, and more. Olostep returns clean, structured data in one single API so that it's compatible with existing backend.

1 request is counted as one webpage/pdf. We don't charge you additionally for GB or for proxies and all those costs are included in the cost per request.

You can access and interact with any website that is publicly accessible. If you are building AI automations and your agent needs to pass cookies or login, get in touch at info@olostep.com

Yes, the API can scale to billions of requests per month.

The API returns the id of the request (for future retrievals), the Markdown and the HTML of the page. You can also retrieve JSON with specific parsers or structured data with LLM extraction. If you are using the /answers endpoint as the search basis for your AI it will return an answer, a json in the schema you have defined and the sources Olostep has searched.

Yes, Olostep lets you extract data using natural language prompts. If you know the exact URL containing your data, use the /scrapes endpoint with llm_extract and describe what you want to extract. At scale, for deterministic needs, we recommend using Olostep's parsers. For more complex tasks like searching for data, navigating between pages, handling pagination, or validating results, use the /agents endpoint that automatically finds and extracts data based on your prompt.

Reach out to us at info@olostep.com or contact our sales team with your use case and we can take a look. Our aim with the Olostep Agent is to be able to automate any business data pipeline and research workflow on the Web so we will do our best to assist your use case.

We don't charge for failed requests. If you are using the answer endpoint or an endpoint that needs to make LLM calls we will pass down those costs to you but on our end we only charge for successful requests.

You can pay using the Stripe Payment Links.

Yes, plans are pro-rated, meaning if you've already paid for a previous plan, the remaining credits will be transferred to your next plan. You won't have to pay again for what you've already covered.

Yes, Olostep is free for the first 500 requests. Then paid plans start from $9/month for 5000 credits per month. Olostep is considered the most reliable and cost effective API on the market. Try it for free and see it on your own.

We're fully committed to building products that you love. If for whatever reason you're unsatisfied with the Olostep API, please email us at info@olostep.com to receive a full refund within a few hours. We'll also refund you if it doesn't turn out being useful. If you decide to use it but only after a certain period of time, we'll refund the time you don't use it.