What Is JavaScript Crawling?

JavaScript crawling finds pages across a site and runs page scripts when needed. The crawler follows allowed links. It renders a page only when raw HTML does not contain the needed content or links.

A crawler starts with one or more seed URLs. It fetches each page, finds links, and adds new URLs to a queue. This basic process explains how web crawlers work for both raw and rendered pages.

JavaScript Crawling Versus Web Scraping

Crawling finds and schedules pages. Scraping pulls content or fields from a page that the crawler has reached.

One pipeline can do both jobs. The crawler expands site coverage. The Olostep Scrape endpoint can then return selected page content in formats such as Markdown, HTML, text, or JSON.

How JavaScript Crawling Works

JavaScript crawling repeats a clear loop. It fetches a URL, renders when needed, finds links, cleans those URLs, queues allowed links, and saves the result.

Google uses its own process for JavaScript pages. Google’s JavaScript SEO documentation states: Google processes JavaScript web apps in three main phases: 1. Crawling 2. Rendering 3. Indexing.

Initial HTML, Background Data, and the Rendered DOM

Initial HTML is the server response before page scripts change the document. It may already contain the text, links, metadata, or data you need.

Pages can use several rendering designs. The web rendering patterns source states: Rendering on the web can happen on the server, on the client, at build time, or through hybrid approaches.

Background JSON or XHR requests may provide a suitable public data source. The rendered DOM is the page structure after JavaScript runs.

A practical check is to inspect the initial response, look for a suitable public data request, and use browser rendering only if the required result still depends on page scripts. The JavaScript rendering guide compares browser rendering with finding backend data APIs directly.

Link Discovery After Rendering

Rendering may reveal links missing from the initial HTML. Menus, result cards, routes, and page controls can add links after scripts run.

Each link still needs a scope check. Resolve relative URLs and apply host and path rules. Reject links outside the planned crawl before adding them to the queue.

When Does a Crawler Need JavaScript Rendering?

A crawler needs rendering when required content, links, or controls appear only after JavaScript runs. Base the choice on the needed result, not the presence of script files.

Use the simplest fetch method that works. Add a browser only when simpler methods cannot return the required state.

Start With the Raw HTTP Response

Check the HTTP response for the required text, links, metadata, or structured data. Use response parsing if that response meets the task.

Do not assume every modern site needs browser rendering. Some pages send useful HTML from the server or use HTML built before deployment. Test the response instead of guessing.

Check JSON and XHR Requests

Inspect public background requests when raw HTML lacks the needed data. A page may request JSON and use it to build the visible screen.

Validate the request against your required fields, target state, and access rules before adding it to the crawl.

Render Only When Content or Actions Require It

Render when JavaScript creates required content, links, routes, page controls, or other page state. Define that state before the browser starts.

Define the target state and verify that the browser reached it. Do not assume one render covers every possible state.

How to Build a JavaScript Crawl Loop

A crawl loop needs a seed list, queue, visited set, and fetch method. It also needs a link finder, scope rules, output store, and failure record.

Production designs also need rules for clean URLs, queue order, stopping, and failed requests. This production crawler architecture explains those parts in more depth.

Normalize and Filter URLs Before Enqueueing

Normalize each URL before checking or queueing it. Resolve relative links, remove fragments, and use one consistent form for the host.

Then apply scope rules. Host, path, include, and exclude rules keep the crawl away from unrelated pages and endless URL patterns.

Use a Frontier and Visited Set

A frontier is the queue of URLs waiting to be fetched. A visited set stops the crawler from fetching the same clean URL again.

The example uses Node.js built-in fetch. It stays on one host and stops after 20 pages. Run it as an ES module in a Node.js version with global fetch.

Node.js crawl-loop example:

const seed = new URL('https://example.com/');
const queue = [seed.href];
const visited = new Set();
const pageCap = 20;

function normalize(href, base) {
  const url = new URL(href, base);
  url.hash = '';
  url.hostname = url.hostname.toLowerCase();
  return url.href;
}

while (queue.length && visited.size < pageCap) {
  const current = queue.shift();
  if (visited.has(current)) continue;
  visited.add(current);

  const response = await fetch(current);
  if (!response.ok) continue;
  const html = await response.text();
  console.log(response.status, current);

  for (const match of html.matchAll(/href=["']([^"']+)["']/gi)) {
    const next = normalize(match[1], current);
    if (new URL(next).host !== seed.host) continue;
    if (!visited.has(next) && !queue.includes(next)) queue.push(next);
  }
}

This code shows the crawl loop. It is not a full HTML parser or a JavaScript renderer. Production code also needs timeouts, content checks, saved queue state, and clear failure types.

Separate Discovery From Extraction

Store crawl records apart from extracted fields. A crawl record can hold the URL, final URL, fetch method, status, found links, and output location.

A later step can turn selected pages into Markdown, text, or schema-based JSON. A schema is a list of expected fields and data types. Olostep reports that its Olostep web scraping API supports rendered retrieval, browser actions, waits, and structured outputs.

How to Wait for JavaScript Pages and Interact With Them

A JavaScript page is ready when the state needed by the task is present. Useful checks include a target element, expected response, URL change, or known app state.

Waiting behavior differs by tool and action. Playwright auto-waiting states: It auto-waits for all the relevant checks to pass and only then performs the requested action.

Choose a Readiness Signal

Choose a signal tied to the output. Wait for an element when a page part must appear. Wait for a response when a known request supplies the data.

A timeout only limits waiting time. After the wait, check the target content or state to confirm success.

Add Clicks, Scrolling, and Pagination Carefully

Add an action only when the needed content or links depend on it. Give every click, scroll, or page step an expected result and stop rule.

Limit infinite scroll by action count, time, new items, or new URLs. The Playwright scraping guide shows waits and browser actions for pages that need them.

How to Scale JavaScript Crawling in Production

Safe scale starts with crawl scope, host health, clear failures, and useful logs. Set request limits from test results and the target’s published guidance.

Google gives one crawler-specific example. Google’s crawl-capacity guidance states: This limits the total amount of time your server spends holding connections open for Google, factoring in both the number of parallel connections and their duration.

Prevent Duplicate URLs and Endless Crawl Loops

Normalize URLs before checking for duplicates. Define rules for fragments, query strings, trailing slashes, redirects, canonical hints, and repeated page states.

Use page and depth caps as backup limits. Stop pagination when it finds no new allowed links. Reject state changes that repeat earlier results.

Control Request Rate and Retries

Set request rates per host. Use measured response time, error patterns, and target guidance instead of one setting for every site.

Retry only failures that may be temporary. Limit each retry count and add a delay. Stop retrying invalid URLs, blocked requests, unsupported files, and repeated failures.

Record Outputs and Failures

Save enough data to explain each result. Useful fields include status, final URL, fetch mode, timing, found-link count, output location, attempts, and error type.

These records support focused retries and repeatable data work. They also let teams compare raw and rendered results without mixing content with job status.

How to Get Started With JavaScript Crawling

Start with one public page, a clear data goal, and a small crawl limit. You also need a Node.js setup or crawl tool, permission to access the target, and a place to save results.

1. Define the result. List the exact text, links, or fields you need. Set allowed hosts, paths, page count, and depth.
2. Test raw HTML. Fetch the seed URL and check for the needed result. If it is present, keep the crawl in HTTP mode.
3. Check background data. Inspect public JSON or XHR requests if raw HTML is incomplete. Use them only when access and output are suitable.
4. Enable rendering. Use a browser when scripts or actions create the required state. Set a clear readiness check and timeout.
5. Crawl within scope. Normalize found links, remove duplicates, and queue only allowed URLs. Stop at the set limits.
6. Verify the output. Confirm the expected fields, status, fetch mode, and found links. Compare a few results with the page in a browser.

The expected result is a small set of crawl records with clear status and content fields. If records are empty, test a longer wait or a better readiness signal. If the queue grows too fast, tighten path rules and page limits.

Keep the first run small. For each rendered step, define the target state and verify that the browser reached it before accepting the result.

Sample Crawl Result (JSON):

{
  "url": "https://example.com/products/widget",
  "status": "success",
  "fetch_mode": "rendered_browser",
  "discovered_links": [
    "https://example.com/products/widget/specs",
    "https://example.com/products/widget/support"
  ],
  "content_format": "markdown"
}

This sample is generic and shows one possible record shape. It is not an exact Olostep API response.

Should You Build a Crawler or Use a Managed API?

The choice depends on the workload. Compare page control, interaction needs, crawl size, job tracking, output formats, and team time.

A managed web crawling API may reduce recurring infrastructure work when its controls fit the crawl. Olostep reports support for async jobs, recursive discovery, rendering, page and depth limits, URL filters, webhooks, and result IDs.

Decision Area	Browser Library	Managed Crawl API
Page control	Direct control through the chosen library	Control through supported API settings
Operations	Team builds and runs the required job system	Provider runs the crawl service
Output	Team defines storage and extraction	Provider may return standard content and result IDs
Best fit	Custom actions on a bounded set of targets	Repeated crawls that fit the API controls

When Browser Libraries Fit

Browser libraries fit when a team needs low-level page control. They also fit when the team can build the needed queue, retry, logging, and browser systems.

The exact work varies by library and design. Test the full operating load before choosing this path for many sites or long jobs.

When a Managed Crawl API Fits

A managed API fits when recursive discovery and browser work are repeated needs. It may also fit when async jobs, scope limits, webhooks, and standard outputs match the pipeline.

Check provider controls against the real workload. Teams still define scope, review results, manage data use, and send content to later systems.

JavaScript Crawling for Google and Technical SEO

Technical SEO tests should compare initial HTML with the rendered DOM. Check content, links, titles, metadata, canonical tags, and page status.

Googlebot also has a specific response-size rule. According to Googlebot’s 2 MB fetch limit, Googlebot currently fetches up to 2MB for any individual URL (excluding PDFs).

Test the Initial and Rendered Versions

Compare both versions for the search signals that matter. The test should show whether key content exists before rendering and whether scripts change it.

Test several page templates. Product pages, category pages, articles, page controls, and client-side routes may not use the same rendering path.

Keep Search-Engine Claims Scoped

Google’s documentation applies to Google Search, so do not treat it as a specification for every crawler.

Treat rendering and indexing as separate checks. Test the pages that matter and verify the final rendered result.

Frequently Asked Questions About JavaScript Crawling

What Is JavaScript Crawling?

In this guide, JavaScript crawling means finding allowed pages and running page scripts only when the required content or links depend on them.

How Is Crawling Different From Scraping?

Use crawling to find and schedule pages. Use scraping to pull content or fields from the pages reached.

When Does a Crawler Need to Render JavaScript?

Choose rendering when the required result still depends on page scripts after checking the initial response and a suitable public data request.

Can Google Crawl JavaScript-Generated Content?

Yes. The Google Search crawling overview states: During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome, similar to how your browser renders pages you visit.

How Do You Crawl a JavaScript Website?

For a JavaScript site, start with a seed URL and the simplest suitable fetch method. Normalize and deduplicate found links before crawling them within scope.

Which Tool Should You Use: Playwright, Puppeteer, Selenium, or an API?

Choose a tool by page control, crawl size, output needs, and the systems your team can run. Do not select one option for every workload.

How Do You Know When a JavaScript Page Has Finished Loading?

Define completion with a check tied to the required result, such as an element, response, URL, or app state. Verify the result after the wait.

How Do You Stop Duplicate URLs or Endless Crawl Loops?

Use normalized URLs, a visited set, path and parameter rules, and page and depth limits. Stop when no new allowed links appear.

How Should You Control Concurrency and Rate Limits?

Set per-host limits from testing and the target's published guidance. Adjust the limits when validation shows overload or access problems.

Is JavaScript Crawling Legal?

Rules depend on the site, data, access method, contracts, location, and purpose, so seek legal advice when needed. The September 2022 Robots Exclusion Protocol, a foundational standard and recency exception, states: These rules are not a form of access authorization.