What is Robots.txt?

The robots.txt file is a small plain text file that lives at the root of your domain-say, https://www.example.com/robots.txt-and provides instructions to automated web crawlers regarding access to your site. It implements the robots exclusion protocol, a convention dating back to the earliest days of the world wide web, and it tells search engines, AI bots, and other web robots which URLs they may or may not crawl.

One file can influence how Googlebot indexes your product pages, how Bingbot discovers your blog, and whether GPTBot, ClaudeBot, or PerplexityBot harvest your content for artificial intelligence training. Yet robots.txt is advisory, not an access-control system. It cannot physically block access or prevent crawlers from visiting a URL-they can simply choose to ignore it.

For Olostep customers building web data extraction pipelines and AI-powered research workflows, understanding robots.txt is essential. It shapes crawler behavior around scraped data sources and signals what content website owners consider off-limits for crawling and training sets. Only 37% of top 10,000 websites had a robots.txt file in 2025, which means many sites leave the door wide open-while others lock it down aggressively.

What robots.txt does:

Guides well-behaved bots on which paths to crawl or skip
Helps conserve server resources and crawl budget
Signals content-use preferences to AI companies

What robots.txt does NOT do:

Physically prevent access to web pages
Act as a security mechanism for sensitive content
Guarantee removal from search results or AI training data

Who should care: SEO teams, web data engineers, AI/ML teams, compliance officers, and anyone managing a public website.

What is a robots.txt file?

A robots.txt file is a plain text file placed in the root directory of a website that implements the robots exclusion protocol. Its job is to tell search engine crawlers and other bots which parts of a site they may crawl. The file must be named robots.txt in all lowercase-no .html extension, no capitalization-and hosted at the root domain (e.g., https://www.example.com/robots.txt). Placing it in a subdirectory like /static/robots.txt means bots will never find it.

Before crawling any web pages on a domain, bots request /robots.txt and parse its txt directives to decide their behavior. If the file is missing or returns a 404, crawlers assume everything is allowed. The protocol originated in 1994 when Martijn Koster proposed it to reduce chaotic crawling on the early internet. For decades it remained a de facto txt standard until September 2022, when RFC 9309 formalized it as an IETF Proposed Standard.

The maximum size for a robots.txt file is 500 kibibytes. Google and other major search engines will ignore content beyond that threshold, so keeping the file concise matters.

How robots.txt interacts with search engine crawlers

Major search engines like Google, Bing, Yahoo, and Yandex generally honor robots.txt rules for crawling. When Google's crawler requests your site, it first fetches /robots.txt, reads the txt rules, then crawls allowed URLs and skips disallowed ones.

A critical distinction: robots.txt governs crawling, not indexing. Disallowed pages may still appear as bare URLs in google search results if discovered via links or an xml sitemap. You might see a URL in search engine results with no snippet or description-because the bot never fetched the page content, but knew the URL existed.

One robots.txt file can contain different rules for different user agents. You can write separate groups for Googlebot, Bingbot, and other user agents, each with its own set of directives. The wildcard * applies to any bot not matched by a named group.

Robots.txt vs robots meta tags vs X-Robots-Tag

These three tools operate at different levels. Robots.txt works at the site or directory level, controlling which paths crawlers may visit. Robots meta tags live inside the HTML <head> of individual web pages (e.g., <meta name="robots" content="noindex, nofollow">). The x robots tag is an HTTP header that applies to PDFs, images, and other non-HTML resources.

Here's the key interaction: if a URL is blocked in robots.txt, crawlers never fetch the page, so they never see any meta tags or x robots tag headers on it. To keep a page out of search results, use a meta robots noindex tag in HTML-but only if the page is actually crawlable.

Tool	Scope	Controls	Requires crawl?
robots.txt	Site/directory	Crawling access	No
Robots meta tags	Per page (HTML)	Indexing, following links	Yes
X-Robots-Tag	Per file (HTTP header)	Indexing, snippets	Yes
For precise control over search results and data exposure, use all three in combination.

Core syntax and directives in a robots.txt file

A robots.txt file is structured as groups. Each group starts with one or more User-agent: lines followed by directives like Disallow: and Allow:. The file must be UTF-8 encoded plain text-no rich formatting. Unrecognized lines are generally ignored by crawlers.

Here's a minimal example:

User-agent: *
Disallow: /admin/
Disallow: /tmp/
Allow: /admin/public-docs/

Sitemap: https://www.example.com/sitemap.xml

User-agent

User-agent identifies which bot the rule applies to. You specify it with User-agent: followed by the bot's token. Common tokens in 2026 include Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot.

Multiple User-agent lines can appear in one file, each starting a new group of directives. Mis-typing a user agent string-like writing Google-bot instead of Googlebot-means that group is silently ignored. Always verify bot names against official documentation from search engine and AI provider sites.

Disallow and Allow

The Disallow directive blocks specific pages or directories from crawling. Disallow: /private/ tells crawlers to skip everything under /private/. The Allow directive permits access to a specific sub-directory even if the parent is disallowed.

User-agent: *
Disallow: /private/
Allow: /private/press-release-2024.html

Paths are relative to the root, not a full url. For Google and most modern crawlers, the longest match wins when Allow and Disallow patterns conflict. A disallow rule on /private/ combined with an Allow on /private/press-release-2024.html means the press release is crawlable while everything else under /private/ stays blocked.

Crawl-delay, Sitemap, and wildcards

Crawl delay is a non-standard directive designed to avoid overloading a web server by spacing out requests. Crawl-delay is ignored by Google but recognized by other crawlers like Bing and Yandex. Google expects you to manage crawl rate through Search Console instead.

The sitemap directive points crawlers to your xml sitemap files. Adding your sitemap URL in robots.txt helps search engines discover your pages. You can include multiple Sitemap: lines:

Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-products.xml

Wildcards in robots.txt allow complex patterns for blocking. Pattern matching can be used in robots.txt for specifying crawl rules-for example, Disallow: /*?sessionid= blocks all parameterized URLs containing session IDs. This is especially useful for e-commerce sites with faceted navigation that generates thousands of duplicate content URLs.

When and why you should use a robots.txt file

Every public website should have at least a minimal robots.txt file, even if it simply allows all bots and lists a sitemap. Use the sitemap directive to guide crawlers to important pages. Without it, crawlers rely solely on links to discover your content.

A robots.txt file can improve SEO by controlling indexing and preventing low-value content from being crawled. But robots.txt is not a security tool for hiding sensitive content. Use password protection on your server for secure content-never rely on disallow lines alone.

SEO and crawl budget use cases

Crawl budget is the number of URLs a search engine is willing to crawl within a given period. Wasting it on login pages, cart flows, or test environments means important pages get crawled less frequently.

Typical patterns to disallow for SEO purposes:

/wp-admin/ and /wp-login.php
/cart/, /checkout/
/search? and other filter URLs
Staging subdomains
Parameter-heavy URLs creating infinite combinations

Robots.txt files can prevent low-value content from being indexed. But avoid blocking CSS or JavaScript files in robots.txt-search engines need these to render and understand your other pages properly. Do not block essential resources like CSS and JavaScript files in robots.txt, or you risk hurting rankings on the pages that rely on them.

Performance, security, and legal risk considerations

Robots.txt can prevent well-behaved bots from hammering resource-intensive endpoints on your server. But malicious bots will ignore disallow rules entirely-and may even use your robots.txt as a roadmap to "interesting" directories.

Robots.txt files are not legally enforceable for crawlers. However, courts increasingly treat them as evidence of notice in scraping and copyright disputes. Robots.txt instructions cannot enforce behavior on crawlers, but disregarding them can weigh against scrapers in litigation-especially when combined with terms of service. Pair robots.txt policies with contracts, API terms, and authentication to create a layered defense.

Robots.txt in the age of AI bots and web data extraction

The explosion of ai bots since 2022 has changed the practical role of robots.txt. In 2023, 306 of the top 1,000 websites blocked GPTBot, and that number has grown. Major publishers have updated their robots.txt to restrict ai companies from harvesting content for training. Research shows that 60% of reputable sites now disallow at least one AI crawler.

Robots.txt files can block AI bots like GPTBot and ClaudeBot, but the trade-off is real: blocking AI crawlers may reduce your visibility in AI-powered answer engines, potentially cutting referral traffic. Meanwhile, some bots-like Bytespider-have been observed ignoring robots.txt entirely, making server-level blocks necessary.

Blocking or shaping access for AI bots

To block specific AI crawlers, create a robots.txt file with separate user-agent groups:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

A partial-control strategy disallows only sensitive directories-like /members/ or /premium/-while allowing crawls of marketing pages. Some AI crawlers also support opt-out via HTTP headers and special meta tags, so align these signals with your robots.txt for consistency.

Robots.txt, web scraping, and Olostep workflows

Compliant scrapers check robots.txt before crawling and adapt accordingly. Olostep customers building large-scale e-commerce or research crawls can read robots.txt at scale to map allowed paths before launching batch requests. Best practices include rate limiting, proper user agent identification, honoring disallow rules, and backing off on HTTP 429 or 503 responses.

Olostep's unified Web Data API simplifies robots.txt-aware crawling through domain mapping and dynamic route discovery that automatically skip blocked areas-treating robots.txt as a first-class constraint in responsible data engineering, not an afterthought.

How to create a robots.txt file from scratch

To create a robots.txt file, follow a simple sequence: decide your crawl policy, write the text file, upload it to the root, and test. This applies whether you run WordPress, Shopify, a custom framework, or a static host. Testing robots.txt syntax with tools prevents accidental blocks of your site.

Step 1: Plan your crawl and indexing strategy

Inventory your site structure: content areas, admin routes, search results pages, and dynamically generated URLs. Build a small matrix mapping URL patterns to desired crawler behavior-which sections should appear in search engines, which should be partially discoverable, and which should stay hidden from crawlers entirely.

Collaborate across SEO, legal, and engineering teams. Decide policies on AI bots, archiving crawlers, and commercial scrapers before writing a single line. This planning reduces the risk of overly restrictive rules that you have to undo after launch.

Step 2: Write and format the robots.txt file

Create the file using a code editor (VS Code, Sublime, or even Notepad). The file must be a plain text file encoded in UTF-8. Here's a practical 2026-style example covering three directories and AI bots:

# Block admin and staging areas
User-agent: *
Disallow: /admin/
Disallow: /staging/
Disallow: /tmp/

# Block AI training bots
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

# Sitemaps
Sitemap: https://www.example.com/sitemap.xml

Use comments starting with # to document why rules exist. Remember that subdomains each need their own robots.txt-blog.example.com and www.example.com are separate origins.

Step 3: Upload, host, and test your robots.txt

Upload the file to your web server's root so it's accessible at https://yourdomain.com/robots.txt. Verify it returns a 200 status code in your browser-not a 404, redirect loop, or 500 error.

Regularly test your robots.txt file for errors using online tools and Google Search Console's robots.txt tester. Retest after any major site migration, HTTPS transition, or domain change. Set a quarterly review schedule to keep the file aligned with evolving site architecture and AI bot policies.

Best practices, common pitfalls, and troubleshooting

Small mistakes in one file can de-index an entire site or leave sensitive content exposed. Here are the essentials:

Do:

Test rules before deploying
Monitor crawler behavior in server logs
Keep the file under 500 KiB
Review quarterly

Don't:

Block CSS/JS needed for rendering
Use robots.txt as a security mechanism
Forget to update after site restructuring
Assume all bots will comply

File location and syntax mistakes

Classic errors include placing the file in a subdirectory, naming it Robots.txt (capital R), or serving it behind redirect chains. Syntax issues like missing colons, using a full url in disallow lines (Disallow: https://example.com/private/), or mixing up Allow/Disallow order can silently break your rules. A robots.txt file can be up to 500 kibibytes in size-but if you're anywhere near that limit, your rules probably need consolidation.

Robots.txt files should be placed in the root directory-always verify this after deployments. Add monitoring that alerts your team if the file returns an error or is unexpectedly empty.

Overly restrictive rules and blocking important content

A single line like Disallow: / under User-agent: * blocks crawling of your entire site. This commonly happens during staging-to-production migrations when a googlebot disallow rule meant for dev gets pushed live. Diagnose blocked content through Google Search Console coverage reports and site: queries.

Removing a Disallow does not guarantee reindexing. Submit sitemaps and request re-crawls of priority URLs to accelerate recovery.

Changes propagate slowly-search engines cache the file and recrawl it periodically, so urgent fixes need to be paired with manual fetch requests.

Monitoring crawler behavior and adapting over time

Use server logs to see which bots hit which URLs, at what rate, and whether they respect your txt directives. Differentiate between good bots (search engines, legitimate AI crawlers) and abusive or unknown crawlers. For the latter, combine robots.txt with rate limiting and firewall rules.

Track pages crawled per day, errors returned to bots, and changes in search results visibility after updates. Olostep's Web Data API exposes detailed telemetry on crawling behavior for customers managing large-scale data pipelines, making it straightforward to align with target sites' robots.txt policies. Continuous adjustment is normal-as your content strategy and AI policies evolve, so should your simple robots configuration.

How Olostep fits into robots.txt-aware crawling and AI data workflows

Olostep is a unified Web Data API that respects robots.txt and helps AI-native teams, data engineers, and SEO stakeholders coordinate compliant crawling at scale. It abstracts away low-level crawling challenges-JavaScript rendering, anti-bot detection, domain mapping-while honoring target sites' robots exclusion rules.

Customers use Olostep to implement agentic research, e-commerce price monitoring, and content enrichment pipelines that begin with reading robots.txt on every target domain. The API outputs structured JSON or Markdown, making it straightforward to feed crawled data into LLMs, analytics systems, or search and discovery applications. Users on the internet building responsible data workflows need tools that treat compliance as a feature, not an obstacle.

Explore Olostep's API and documentation to build robots.txt-aware web data workflows that are both powerful and responsible. Whether you're extracting product data, enriching AI training sets, or monitoring competitor pages, start with a crawling infrastructure that respects the rules of the sites you depend on.