How to Find All URLs on a Website: 6 Methods

Finding every URL on a website is one of the first steps in a site audit, content inventory, or competitive research project. This guide walks through six methods, from quick free options to full-scale API-driven crawls, so you can pick the approach that fits your job.

The Quick Answer: How to Find Every URL on a Website

You can find all URLs on a website using one of six main methods:

1. XML sitemap — Download the site's /sitemap.xml file.
2. robots.txt — Check for sitemap references and disallowed paths.
3. Google site: operator — Search site:example.com to see indexed pages.
4. SEO crawler — Run a tool like Screaming Frog to crawl and export URLs.
5. Custom script — Write Python or Node.js code to fetch pages and parse links.
6. Crawling and mapping API — Send one request that combines sitemap discovery, robots.txt, and a JavaScript-rendering crawl, then returns a deduplicated URL list.

The best method depends on site size, whether pages rely on JavaScript, and whether you need a one-time export or a repeatable pipeline. For a quick, no-code option, a crawling API handles multi-signal discovery in a single call.

What Does It Mean to Find All URLs on a Website?

A URL (Uniform Resource Locator) is the address of a single page on the web. When you want to "find all URLs on a website," you're building a complete list of every page that lives on one domain—not just the links visible on the homepage.

This is different from extracting outbound links from a single page. A full-site inventory starts at the root and discovers every internal page, whether it's in the navigation, buried five clicks deep, or only reachable through a sitemap. Understanding how a web crawler works helps explain why: a crawler starts at a seed URL, follows internal links, and repeats until it runs out of new pages to visit.

Readers typically fall into two groups. Some need a quick one-off list they can export and move on. Others need a repeatable, programmatic process that handles thousands (or millions) of URLs on a schedule. The methods below cover both.

Method 1: Check the Website's XML Sitemap

An XML sitemap is a file that lists URLs a site wants search engines to find. It usually lives at example.com/sitemap.xml or is referenced inside the site's robots.txt file. Understanding sitemaps in web crawling clarifies why this method is fast: the site has already done the inventory work for you.

To use it, open the sitemap URL in a browser or fetch it with a command like curl. Copy the URLs or save the file for processing. This is the fastest free method when a sitemap exists and is well-maintained.

Key point: Sitemaps have size constraints. According to Google sitemap size limits, a single sitemap file is limited to 50 MB uncompressed or 50,000 URLs, and larger sites must split into multiple sitemaps. If a site uses a sitemap index (a file that points to other sitemaps), you'll need to fetch each child sitemap separately. Per Google sitemap guidance, sitemaps help discovery but don't guarantee every page is crawled or indexed—so a sitemap alone is rarely 100% complete.

Find the Sitemap in robots.txt

The robots.txt file (found at example.com/robots.txt) often contains a Sitemap: line that points to the sitemap's location. It also lists Disallow: rules that tell crawlers which paths the site asks them not to access.

Checking robots.txt gives you two pieces of information at once: where the sitemap lives and which sections the site prefers to keep off-limits. Always respect robots directives when crawling.

Method 2: Use Google's site: Search Operator

The Google site: operator lets you see which pages Google has indexed for a domain. Type site:example.com in Google Search to get a list of results.

You can refine the query to target sections (site:example.com/blog) or file types (site:example.com filetype:pdf). This is useful for quick reconnaissance without installing anything.

Key point: The results show only indexed pages, not every page that exists. According to Google site: operator limits, the returned list is not always exhaustive, and bigger sites shouldn't expect to see all their URLs. New pages, pages blocked by robots.txt, and pages that failed indexing won't appear.

Method 3: Run an SEO Crawler (e.g., Screaming Frog)

Desktop SEO crawlers (a crawler is software that systematically visits pages and follows links) start at a homepage, follow internal links, and build a URL list as they go. Screaming Frog, Sitebulb, and similar tools fall into this category.

These crawlers export to CSV or Excel, making them popular for audits. They also flag broken links, redirects, and missing metadata in the same run.

Key point: Free tiers often cap crawls at 500 URLs. The crawl runs on your machine, so large sites take time and bandwidth. By default, most desktop crawlers fetch raw HTML and may miss links that only appear after JavaScript executes.

Method 4: Write a Custom Script (Python or Node.js)

A custom script gives you full control. The typical pattern:

1. Fetch a page's HTML (e.g., with Python's requests or Node's axios).
2. Parse the HTML for <a href> tags (BeautifulSoup in Python, Cheerio in Node.js).
3. Filter for internal links (same domain).
4. Add new URLs to a queue and repeat until the queue is empty.

This approach is flexible and free. You can crawl subpages automatically by recursing through internal links.

Key point: DIY scripts require maintenance. Sites block repeated requests, layouts change, and JavaScript-rendered content won't appear in the raw HTML. Handling proxies, retries, and browser rendering adds engineering work that grows with scale.

Method 5: Use a No-Code URL Extractor Tool

Online URL extractors let you paste a domain and download a URL list without writing code. Most export to CSV, TXT, or JSON.

Olostep's free URL extractor tool uses the Maps endpoint to discover a site's structure via sitemaps, robots.txt, and heuristic scanning, then deep-crawls with cloud browsers that render JavaScript. The result is a deduplicated, normalized list in JSON, CSV, or TXT.

Key point: No-code tools are best for quick, one-off jobs. Check whether the tool renders JavaScript (many don't) and whether there's a row cap before relying on it for large sites.

Method 6: Get Every URL in One Call with a Crawling and Mapping API

A crawling and mapping API handles the entire discovery process in a single request. Olostep's /crawls endpoint starts at a root URL, follows internal links to a configurable depth and scope, and works even when the target site has no sitemap. You can get all pages of a website by combining sitemap discovery, robots.txt parsing, and a real-browser crawl that renders JavaScript—then receiving deduplicated, normalized URLs in JSON, CSV, or TXT.

Here's a Python example that starts a crawl:

import requests

API_URL = 'https://api.olostep.com/v1/crawls'
API_KEY = '<token>'

headers = {'Authorization': f'Bearer {API_KEY}'}
data = {
    "start_url": "https://docs.stripe.com/api",
    "include_urls": ["/**"],
    "max_pages": 10
}

response = requests.post(API_URL, headers=headers, json=data)
result = response.json()
print(result['id'])

This code sends a POST request to start a crawl at docs.stripe.com/api, includes all subpaths, and caps the crawl at 10 pages. The API returns a job ID you can poll for results.

Key point: Every Olostep request is JavaScript-rendered and uses residential IPs with proxy rotation, so you get the same page a real browser sees without managing infrastructure yourself.

Why Single Methods Miss Pages (JavaScript, Orphan, and Hidden URLs)

Sitemaps, the site: operator, and basic scripts each have blind spots. Combining them closes the gap.

JavaScript-rendered pages — Many modern sites load content dynamically. According to W3Techs JavaScript usage data, JavaScript is used as a client-side language by 98.9% of all websites. As Google JavaScript SEO basics explains, "Some JavaScript sites may use the app shell model where the initial HTML does not contain the actual content and Google needs to execute JavaScript before being able to see the actual page content that JavaScript generates." A raw HTML fetch misses those links entirely.

Orphan pages — An orphan page is a page with no internal links pointing to it. Because crawlers follow links, orphan pages won't appear unless they're in a sitemap or you know the URL directly.

Login-gated and hidden pages — Pages behind authentication or pages intentionally excluded from sitemaps won't show up in public methods. (Never bypass logins or access controls when crawling.)

A real-browser crawl that combines sitemap parsing, robots.txt awareness, and JavaScript rendering recovers many of these missed URLs.

Discovering URLs at Scale and on a Schedule

For many domains or recurring audits, batch discovery and scheduled re-crawls beat manual runs.

Olostep supports bulk URL extraction that can scrape approximately 100,000 pages in 5–7 minutes. With up to five concurrent batch threads, the throughput scales to approximately 1,000,000 requests in around 15 minutes.

For ongoing monitoring, scheduled research agents let you define a workflow in plain language and run it on a cadence (e.g., weekly). New URLs or changes trigger notifications and export to Google Sheets, CSV, or JSON—useful for competitive intelligence or content-change tracking.

Key point: Structured JSON output fits directly into AI/RAG pipelines, so the URL list becomes an input for downstream automation without manual reformatting.

How to Choose the Right Method

Use this table to match your situation to the right approach:

Key point: For small static sites, the sitemap or site: operator is often enough. For completeness on JavaScript-heavy sites, recurring jobs, or large-scale pipelines, a crawling and mapping API is the most practical choice.

What to Do With Your List of URLs

A complete URL list unlocks several workflows:

• SEO and site audits — Find broken links, redirect chains, duplicate content, and pages missing metadata.
• Content inventory — Catalog every page before a redesign, migration, or CMS switch.
• Orphan page discovery — Identify pages with no internal links so you can add navigation or consolidate content.
• Competitive intelligence — Map a competitor's site structure to see what they publish and how they organize it.
• AI and LLM workflows — Build a crawl list for RAG pipelines, AI-visibility audits, or training-data collection.

Why does completeness matter? Organic clicks are scarce: a SparkToro zero-click search study found that ~68% of US Google searches ended without a click in the first four months of 2026 (based on Similarweb clickstream data measuring search sessions, not individual pages). With clicks concentrated on a small share of pages, a complete URL inventory shows you exactly which pages earn traffic—and which to fix or consolidate.

Frequently Asked Questions

How Do I Find All Pages on a Website for Free?

Use the site's /sitemap.xml, Google's site: operator, or a free online URL extractor. Each is free, but may not capture every page if the site uses JavaScript or lacks a complete sitemap.

What If a Website Has No Sitemap?

Crawl it from the homepage by following internal links. An SEO crawler, a custom script, or a crawling API can all discover pages without relying on a sitemap.

Why Are Some URLs Missing from My Results?

Pages can be blocked by robots.txt, hidden behind logins, generated by JavaScript, or orphaned with no internal links. A single method often under-counts; combining methods improves coverage.

What's the Difference Between Indexed Pages and All Pages?

Indexed pages are only those a search engine has stored in its database. A site can have many more live pages that were never indexed because they're new, blocked, or undiscoverable.

How Do I Export the List of URLs?

Most tools and APIs export to CSV, TXT, or JSON. JSON is best for feeding pipelines, agents, or spreadsheets that need structured input.

Can I Find All URLs on a Competitor's Website?

Yes, using public methods like their sitemap, the site: operator, or a crawler. Always respect the site's terms, robots directives, and rate limits.