What is crawl delay?

Crawl delay is a request in a robots.txt file that asks a crawler to fetch pages more slowly. The RFC 9309 robots standard does not define crawl-delay, and it states, “The parsing limit MUST be at least 500 kibibytes.” The field remains optional and meaningful only when a specific crawler documents support.

A delay can be one input to crawler policy, but it is not a complete rate-control system. To see where it fits, first understand how web crawlers work across URL discovery, scheduling, fetching, parsing, deduplication, and recrawling.

What Does Crawl-Delay: 10 Mean?

Crawl-delay: 10 means ten seconds only under the supporting crawler's documented timing model. It does not create one universal rule for every web crawler.

For Bingbot, a value of 10 allows at most one URL in each ten-second window. Another crawler could interpret the value differently or ignore it, so you cannot safely convert it into a universal six-requests-per-minute rule.

Actual throughput can also fall below the theoretical ceiling because requests may fail, time out, or wait in a queue. In an illustrative multi-worker model, independent delays can still produce overlapping requests and a higher aggregate rate.

How Does Crawl Delay Work in Robots.txt?

A crawler that supports crawl-delay reads the robots.txt group matching its user agent and applies its own timing rules. The file declares a site owner's crawl preference, while the crawler's scheduler decides how to execute that preference.

Robots.txt applies to the host, protocol, and port where the file is published. Compliance remains voluntary because robots.txt communicates instructions rather than enforcing them at the network or application layer.

Crawl-Delay Syntax Example

A crawl-delay rule places the field inside the intended crawler's user-agent group. This example asks ExampleBot to use a value of ten seconds:

User-agent: ExampleBot
Crawl-delay: 10

The rule has an effect only if ExampleBot currently documents support for this field. Using User-agent: * does not make the behavior interoperable across crawlers.

Why Timing Semantics Differ

Timing semantics differ because the standardized Robots Exclusion Protocol does not define crawl-delay. A supporting crawler must define whether the value controls request starts, completed responses, or fixed time windows.

For a value of ten, three possible models produce different schedules:

• Start to start: Request A starts at 0 seconds, and request B may start at 10 seconds.
• End to start: Request A ends at 3 seconds, and request B may start at 13 seconds.
• Fixed window: The crawler may fetch one URL during each ten-second window.

Concurrency adds another source of variation. In an illustrative four-worker model, separate ten-second waits can overlap and create bursts at the host.

Robots.txt Is Not Authorization or Security

Robots.txt is not authorization, authentication, or access control. It cannot protect sensitive URLs, grant legal permission, or replace server-side security.

A robots.txt rule also does not guarantee deindexing. A crawler may avoid fetching a blocked URL while still learning about that URL through links or other sources.

Which Crawlers Support Crawl Delay?

The table below summarizes this article’s crawler-support findings. The Googlebot and Bingbot sections provide the supporting details and current first-party sources.

Use this source order when claims conflict: current crawler documentation, current standards, infrastructure documentation, product documentation, then legacy or community material.

Crawler	Support Status	Documented Timing Model	Primary Source	Best Alternative or Next Step
Googlebot	Unsupported	No crawl-delay interpretation	Current Google crawler documentation	Use Google's documented overload guidance and official support paths.
Bingbot	Supported	One URL per 1–20 second window	Current Bing Webmaster guidance	Set a supported value, then verify observed Bingbot traffic.
Any unverified crawler	Unknown	Unknown until documented	The crawler owner's current documentation	Check the documentation or contact the crawler operator.

Googlebot Does Not Support Crawl Delay

Googlebot does not support the crawl-delay field. The current Google source and parser limit appear below.

The Google robots.txt specification also states, “Google enforces a 500 kibibyte (KiB) robots.txt file size limit. Content which is after the maximum file size is ignored.” Place important supported rules before that limit, but do not expect crawl-delay to affect Googlebot.

Bingbot Uses Time Windows

Bingbot interprets crawl delay as a fixed window with at most one fetched URL. The Bingbot crawl-delay guidance describes “small windows (1 to 20 seconds), during which Bingbot can crawl only one URL”.

With Crawl-delay: 5, Bingbot can fetch at most 17,280 URLs per day if it uses every five-second window. That number is a Bingbot-specific capacity ceiling, not guaranteed throughput or a formula for other crawlers.

Treat Unverified Crawlers as Unknown

As a verification practice, treat a crawler’s support as unknown until current first-party documentation confirms the field and timing model. A blog post, old support answer, or observed pause can raise a question, but it does not establish present behavior.

A practical verification step is to check the crawler’s exact user-agent token, supported syntax, allowed value range, and timing definition. Then compare the published rule with server logs to assess observed behavior.

When Should You Use Crawl Delay?

Use crawl delay when a named crawler documents support and a fixed crawler-specific cap matches your goal. Choose another control when you need precise, dynamic, or coordinated traffic management.

The decision depends on server pressure, crawler support, freshness needs, and the controls available on both sides. There is no universal safe delay for every site or workload.

When Crawl Delay Can Help

Crawl delay can help reduce traffic from a supporting named crawler when that traffic strains a host. Scope the rule to the intended user agent instead of assuming every crawler will interpret a wildcard group alike.

Measure request starts, response times, and errors before and after the change. Keep the rule only if observed traffic follows the documented model and server health improves.

When Crawl Delay Can Hurt Freshness

A larger supported delay lowers the crawler's possible fetch capacity and may slow content discovery or refresh. The effect depends on site size, update frequency, queue priorities, and whether the crawler honors the rule.

Treat the setting as a capacity and freshness trade-off, not a ranking control. If important pages change often, reduce unnecessary URLs before slowing every eligible fetch.

When to Use Another Control

Use another control when crawl-delay is unsupported, traffic conditions change quickly, or several workers share the same host. Server responses can signal temporary pressure, while crawler queues and host budgets control client-side request scheduling.

For an emergency Googlebot overload, Google recommends temporary 429 or 503 responses. The Google crawling error guidance states, “Googlebot will retry these URLs for about 2 days.” Continuing those responses beyond about two days can cause affected URLs to be dropped from Google's index.

Why Fixed Crawl Delay Is Not Production Crawler Pacing

Fixed crawl delay controls one timing value, while production load depends on concurrency, retries, latency, endpoint cost, and URL volume. A reliable crawler needs a feedback loop that changes pressure when the target or workload changes.

A ten-second sleep may look cautious in one worker and still create a burst across a distributed fleet. It also cannot express that a search page costs less to serve than a rendered report or large file.

Concurrency Changes the True Request Rate

A practical implementation should calculate host pressure from aggregate worker traffic. Per-worker sleep alone should not be treated as a host-wide limit.

In an illustrative four-worker model, ten-second spacing per worker could produce about 24 request starts per minute. This calculation ignores response timing, retries, restarts, and burst alignment, so it is not an operating target.

Distributed Workers Need Per-Host Coordination

A practical implementation should coordinate workers targeting one origin through a per-host scheduler or token budget. That policy can govern request starts, active concurrency, cooldowns, and recovery across processes.

A practical implementation can isolate rate state by host so one target’s throttling does not stall unrelated work. Olostep documents batch URL processing for large URL lists, parallel batches or workflows, and per-item result retrieval.

Server Policies Can Be Multidimensional

Server policies can vary by host, path, identity, session, operation, or resource cost. One robots.txt number cannot describe all those dimensions or adapt when server capacity changes.

A crawler should therefore combine host-level pacing with response-aware rules. Expensive endpoints may need lower concurrency, while lightweight endpoints may remain healthy at a different rate.

How Should a Production Crawler Pace Requests?

A production crawler should combine cautious per-host concurrency, explicit server feedback, bounded retries, jittered backoff, efficient URL selection, and measurable recovery rules. Olostep says its managed web crawling API handles queueing, rate limiting, retries, robots.txt behavior, and crawl-scope controls.

The base delay, concurrency ceiling, retry cap, and recovery ramp remain application policy choices. Set them from target guidance and observed behavior rather than copying a universal preset.

Handle 429, 503, and Retry-After

The HTTP 429 standard states, “The 429 status code indicates that the user has sent too many requests in a given amount of time.” Treat 503 as a temporary unavailability signal, and use a valid Retry-After value to set the earliest retry time under your crawler policy.

The HTTP Retry-After standard explains, “Servers send the Retry-After header field to indicate how long the user agent ought to wait before making a follow-up request.” The header can express a delay in seconds or an HTTP date, but the server cannot guarantee every client will comply.

Signal	Likely Meaning	Immediate Crawler Action	Recovery Condition
429 Too Many Requests	The client exceeded a rate policy	Honor valid Retry-After, lower host concurrency, and delay retries	Resume slowly after the wait and a run of healthy responses
503 Service Unavailable	The service is temporarily unavailable or overloaded	Honor valid Retry-After, pause or reduce pressure, and avoid retry storms	Resume after the wait when latency and status codes recover
Rising latency without 429/503	The host may be approaching capacity	Reduce request starts and watch error trends	Increase slowly only after latency returns to the healthy range

Use Exponential Backoff With Jitter

Use bounded exponential backoff with jitter when a retryable failure has no usable Retry-After value. The AWS retry behavior guide states, “Standard mode retries failed requests using exponential backoff with jitter.”

The following pseudocode coordinates retries with lower host concurrency. Its delay values and retry count are illustrative policy settings, not HTTP or robots.txt standards.

async def fetch_with_pacing(url, host_state, policy):
    for attempt in range(policy.max_attempts):
        await host_state.wait_for_slot()
        response = await fetch(url)

        if response.status not in {429, 503}:
            host_state.record_success(response.latency)
            return response

        host_state.reduce_concurrency()
        retry_after = parse_retry_after(response.headers.get("Retry-After"))

        if retry_after is not None:
            wait_seconds = retry_after
        else:
            backoff_cap = min(policy.max_backoff, policy.base_delay * (2 ** attempt))
            wait_seconds = random.uniform(0, backoff_cap)  # full jitter

        host_state.pause_until(now() + wait_seconds)

    raise RetryLimitReached(url)

Jitter spreads retries across time so workers do not wake together after a shared failure. Recovery should also be gradual, with host concurrency increasing only after a policy-defined run of healthy responses.

Reduce Requests Before Tuning Delay

Reduce unnecessary requests before tuning delay values. Sitemaps, canonical URL selection, deduplication, include and exclude rules, page limits, and depth limits can remove work without reducing useful coverage.

Change-aware recrawling can avoid repeated fetches of stable pages. Olostep's scheduled website change alerts support scheduled checks that can feed update-driven workflows instead of aggressive full recrawls.

Monitor Whether the Rate Is Safe

A safe rate keeps the target healthy while the crawl makes useful progress. Monitor both server feedback and job behavior because a low error count can still hide slow queues or wasteful duplicate fetches.

Track these signals and connect each one to an action:

• Response status: Slow or pause when 429 and 503 rates rise.
• Latency: Reduce pressure when median or tail latency moves above the healthy baseline.
• Retries: Inspect repeated failures and stop retries at the policy limit.
• Host concurrency: Compare active requests with the configured host budget.
• Queue age: Adjust scope or capacity when useful URLs wait too long.
• Duplicate fetches: Fix normalization and deduplication before adding throughput.
• Transferred bytes: Lower scope or concurrency when heavy responses strain either side.
• Freshness: Change revisit schedules when data becomes stale before the next successful crawl.

Operational Checklist for Respectful Crawling

A respectful crawl starts with verified target guidance, conservative host controls, and clear stop conditions. The same controls must remain observable during execution and reviewable after completion.

Use the following checklist as an operating procedure. Record policy choices per host rather than treating them as global defaults.

Before the Crawl

Before the crawl, define what you will fetch and which rules apply. A narrow, verified scope reduces load before pacing begins.

1. Confirm that each target is within the job's approved scope.
2. Inspect robots.txt and any published crawler guidance for the target host.
3. Identify your crawler with a stable user agent and contact information when appropriate.
4. Verify crawl-delay support through current first-party documentation for the named crawler.
5. Set conservative per-host concurrency, retry limits, and a kill switch.
6. Limit URLs with sitemaps, canonical rules, patterns, depth, page count, and deduplication.
7. Define stop conditions for repeated 429/503 responses, rising latency, or unexpected scope growth.

During the Crawl

During the crawl, enforce one shared host policy across all workers. Metrics should trigger actions rather than sit unused in a dashboard.

1. Coordinate request starts and active concurrency per host.
2. Watch response classes, latency, queue age, retry counts, and transferred bytes.
3. Honor valid Retry-After values after 429 or 503 responses.
4. Reduce concurrency and add bounded jittered backoff after repeated pressure signals.
5. Pause the host when failure or latency thresholds cross the defined stop condition.
6. Resume gradually only after the recovery condition is met.

After the Crawl

After the crawl, reconcile every input URL with a completed, skipped, or failed outcome. Retry only failed or missing items instead of rerunning the full job.

1. Match results to stable URL or job identifiers.
2. Inspect host errors, retry exhaustion, duplicate fetches, and out-of-scope discoveries.
3. Validate that pacing controls changed behavior after pressure signals.
4. Adjust host concurrency, retry policy, scope, or revisit schedules from the observed results.
5. Schedule future crawls around actual content change rates and freshness needs.

How Does Crawler Pacing Connect to Structured Output?

Crawler pacing governs when pages are retrieved, while extraction determines whether the retrieved data is useful. Rendering, parsing, and validation convert responses into Markdown, text, or schema-shaped fields for an application.

Olostep's structured JSON extraction maps page content into requested fields after retrieval. This downstream step does not replace pacing, because useful output still depends on controlled and successful access to the source.

Crawling and Scraping Are Separate Stages

In this article, crawling means discovering and scheduling URLs, while scraping means retrieving content from a selected URL. Parsing then maps that content into documents or structured fields that an AI or data pipeline can consume.

This creates four practical layers: access reaches the page, control schedules the request, extraction creates the output, and the application uses it. A responsible web scraping API addresses retrieval and extraction for known URLs, while a crawler adds discovery and site-wide scheduling.

Managed Infrastructure Versus In-House Controls

Managed infrastructure fits workloads with many targets, changing site behavior, browser rendering, structured output, and strong observability needs. In-house controls can fit narrow, stable, low-volume targets when one team owns the scheduler, retries, parsing, and on-call response.

Compare both options across target count, workload variability, browser needs, retry isolation, host coordination, output schema, monitoring, and engineering ownership. Teams remain responsible for approved scope, source rules, and how they use the resulting data.

Crawl Delay FAQ

These recommendations help teams turn crawl-delay guidance into an operating policy without repeating the crawler-specific facts above.

How Should I Choose a Crawl-Delay Value?

Start with the named crawler’s documented range and the target host’s observed capacity. Adjust only after comparing request traffic, latency, errors, and freshness.

What If Several Workers Crawl the Same Host?

A practical implementation should coordinate request starts and concurrency through one host policy. Treat worker-count calculations as illustrative until server logs confirm the resulting traffic pattern.

When Should I Stop Retrying?

Stop at your application’s defined retry limit or earlier when repeated pressure signals, latency, or scope conditions trigger a kill switch. Record the final outcome instead of restarting the entire job.

When Does Managed Crawling Make Sense?

Managed crawling can fit workloads where target count, rendering, variability, retries, monitoring, or structured output create sustained operational work. In-house controls can fit narrow, stable jobs with clear engineering ownership.