What Is a Sitemap?

A sitemap is a file that gives search engines information about important site URLs and files. The Google sitemap overview states, “A sitemap is a file where you provide information about the pages, videos, and other files on your site.”

This guide focuses on sitemap formats that Google supports. For an audit, treat the sitemap as the site’s declared URL list, then compare it with URLs found through a permitted crawl.

What Sitemap Formats Does Google Support?

Google supports XML, RSS, mRSS, Atom, and text sitemap formats. XML also supports extensions that provide details about images, video, and news content.

HTML sitemap pages can still help users and crawlers follow internal links. However, Google does not list HTML as a submitted sitemap format.

XML, RSS, Atom, and Text Sitemaps

XML is the most flexible supported format because it can include URL metadata and content extensions. RSS, mRSS, and Atom feeds usually cover recent URLs and can fit publishing systems that already create feeds.

A text sitemap contains one absolute URL per line. It cannot include the extra metadata available in XML.

An HTML sitemap is an ordinary page that organizes internal links for visitors. Its discovery value comes from crawlable links, not a special sitemap submission format.

Google's Google internal link guidance says, “Every page you care about should have a link from at least one other page on your site.” An HTML navigation page can support that goal when its links are useful and accessible.

Image, Video, and News Extensions

XML sitemap extensions can provide extra information about images, video, and news content. Use an extension when its fields match content that you want Google to understand.

These extensions add details to XML entries. They do not create a separate guarantee that Google will crawl, index, or rank the listed content.

What Does a Sitemap Help With?

A sitemap helps search engines discover URLs and understand selected details about them. It complements internal navigation and other discovery routes.

Discovery, crawling, indexing, and ranking are separate processes. A search engine may discover a URL without crawling or indexing it.

When a Sitemap Is Most Useful

Google highlights sitemaps for new sites, large or complex sites, and sites with substantial image, video, or news content. Smaller sites with comprehensive internal links may need less sitemap support.

Teams can also use a sitemap as an operational inventory. For example, a team may compare it with crawl results during a migration or audit.

What a Sitemap Does Not Guarantee

A sitemap does not guarantee crawling, indexing, or ranking. It tells a search engine about URLs, while later processing depends on the platform and each page.

A sitemap also does not prove that an audit found every route. Coverage depends on the declared files, crawl scope, permissions, page behavior, and observation time.

What Does an XML Sitemap Look Like?

An XML sitemap places each absolute URL inside a <url> entry. The required <loc> element contains the preferred URL.

The file may also include an accurate <lastmod> value for a significant page change. The examples below use copy-pastable structures.

Annotated XML Sitemap Example

A basic XML sitemap contains a declaration, a <urlset> root, and one <url> block per page. This example lists two preferred URLs.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-08-20</lastmod>
  </url>
  <url>
    <loc>https://example.com/guides/sitemaps</loc>
    <lastmod>2026-08-27</lastmod>
  </url>
</urlset>

XML declaration: The first line identifies the XML version and UTF-8 encoding.
URL set: The <urlset> element contains the URL entries and declares the sitemap namespace.
URL entry: Each <url> element groups data about one page.
Location: Each <loc> value contains a fully qualified URL with its protocol and host.
Last modified: Use <lastmod> only when it reflects a verified, significant page change.

Google ignores <priority> and <changefreq> values. Do not use those elements to predict Google's crawl order or frequency.

Sitemap Index Example for Multiple Files

A sitemap index lists several sitemap files under one discoverable URL. Large sites can use indexes to split inventories into smaller, owned files.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://example.com/sitemaps/products.xml</loc>
    <lastmod>2026-08-28</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://example.com/sitemaps/articles.xml</loc>
    <lastmod>2026-08-27</lastmod>
  </sitemap>
</sitemapindex>

Each <sitemap> block points to one child file. The Sitemaps protocol states, “A Sitemap index file may not list more than 50,000 Sitemaps.”

How Do You Find a Website's Sitemap?

You can find a sitemap by checking common paths, robots.txt, sitemap indexes, CMS settings, and verified search tools. Follow indexes because one sitemap location may lead to several child files.

Check https://example.com/sitemap.xml.
Check https://example.com/sitemap_index.xml.
Open https://example.com/robots.txt and look for Sitemap: lines.
Follow each sitemap index to its child files.
Check the site's CMS settings or documented conventions.
Review the Sitemaps report if you have verified Search Console access.
Use a sitemap URL extractor for programmatic extraction from sitemap files and indexes.

The expected result is a list of sitemap files and their declared URLs. A failed common path does not prove that the site lacks a sitemap.

Common Sitemap Locations

Common locations include /sitemap.xml, /sitemap_index.xml, and paths declared in robots.txt. These paths are conventions, not universal requirements.

A site may keep several files under a /sitemaps/ directory. It may also separate files by content type, locale, or another stable boundary.

Robots.txt Sitemap Directive Example

The example below shows a Sitemap: line with an absolute sitemap URL.

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Olostep’s robots.txt protocol guide provides a practical syntax walkthrough.

How Do You Create a Sitemap?

Create a sitemap by choosing a method, selecting preferred URLs, generating a supported format, and publishing the file. The method should match the site's size and architecture.

A valid file still needs a URL-quality review. Check status, canonical intent, scope, and access before submission.

Choose a Creation Method

Choose a method based on inventory size, maintenance ownership, and site changes. Google recommends automatic generation for sitemaps with more than a few dozen URLs.

Method	Best Fit	Maintenance Model	Scale	Main Risk
CMS-generated	Sites managed by a supported CMS	Updates during publishing	Small to large	Configuration errors
Generator-based	Small, mostly static sites	Rerun after meaningful changes	Small to medium	Stale output
Manual XML	Tiny, stable sites	Edit and validate by hand	Very small	Syntax errors
Crawler or API-supported	Custom or changing sites	Scheduled discovery and generation	Medium to large	Unclear crawl scope

WordPress and other content systems may create sitemaps automatically. Review their output instead of assuming the defaults match your URL policy.

Select Which URLs to Include

Include fully qualified, preferred canonical URLs intended for search results. Each URL should use the chosen protocol, host, path, and casing.

Exclude duplicates, redirects, errors, and noncanonical variants. Also exclude pages intentionally kept out of search results.

Generate and Publish the File

Generate the file in a Google-supported format and publish it at an accessible site URL. Keep each listed URL within the applicable host and path scope.

Request the sitemap URL directly after publication. Confirm that the response contains the intended file and does not return an unexpected redirect or error.

How Do You Submit a Sitemap to Search Engines?

Submit a sitemap through the search engine's supported tools or discovery routes. Submission provides the file location but does not guarantee crawling or indexing.

The official Bing sitemap guidance states, “Up to 50,000 URLs per individual sitemap file.” This limit does not promise indexing or visibility for those URLs.

Submit and Monitor in Google Search Console

Submit a sitemap through the Sitemaps report in Google Search Console. You need verified access to the relevant property.

Open the property in Search Console.
Open the Sitemaps report.
Enter the sitemap path or URL.
Submit the file.
Review fetch status and processing errors.
Correct reported issues and verify the updated file.

A successful request confirms that Search Console received the submission. It does not confirm that Google will crawl or index every listed URL.

Reference the Sitemap in Robots.txt

Add an absolute Sitemap: URL to robots.txt when you want crawlers to discover the file there. This route remains separate from robots.txt access rules.

Robots.txt manages crawler access rather than reliable search-result exclusion. The Google robots.txt guidance states, “It is not a mechanism for keeping a web page out of Google.”

Use a suitable indexing or access control when exclusion is the goal. The correct method depends on whether a crawler may retrieve the page.

How Do You Validate and Maintain a Sitemap?

Validate a sitemap with a practical checklist derived from the protocol and current Google guidance. Check both file correctness and the intent of every listed URL.

A syntax checker cannot confirm canonical intent or page status. Combine file-level validation with page-level checks and search-engine processing reports.

Sitemap Validation Checklist

Use this checklist before submission and during scheduled audits. Record failures so the team can verify each correction.

Response status: Confirm that the sitemap returns HTTP 200 without an unexpected redirect.
Supported format: Use XML, RSS, mRSS, Atom, or text according to the receiving platform's guidance.
XML syntax: Check for malformed tags, bad escaping, or invalid characters in XML files.
Namespace: Confirm that an XML root element uses the correct sitemap namespace.
Absolute URLs: Include the protocol, host, and complete path in every <loc> value.
Allowed scope: Keep URLs within the sitemap's permitted host and path scope.
Canonical intent: List the preferred canonical version of each page.
Duplicates: Remove duplicate URLs and normalized variants.
Redirects and errors: Remove URLs that redirect or return error responses.
Indexing intent: Exclude pages intentionally kept out of search results.
Modification dates: Include <lastmod> only when the value is accurate.
Processing reports: Review platform reports for fetch or parsing problems.

The expected result is an accessible file containing intended canonical URLs. Investigate each mismatch before treating the inventory as ready.

File Limits and Google Creation Rules

One sitemap file is limited to 50,000 URLs or 50 MB uncompressed. The Google sitemap limits state, “All formats limit a single sitemap to 50MB (uncompressed) or 50,000 URLs.”

The same guidance requires UTF-8 and fully qualified, absolute URLs. It recommends preferred canonical URLs and accurate <lastmod> values for significant changes.

Google supports submission through Search Console, the Search Console API, and robots.txt. Split larger inventories into multiple files and list them in a sitemap index.

Update Frequency and Metadata Accuracy

Update a sitemap when its preferred URL inventory changes. Regenerate it when verified modification data must change, not just to create a newer timestamp.

Record the event that triggered each update. This makes stale files and false modification dates easier to trace.

How Should Large and Dynamic Sites Manage Sitemaps?

Large and dynamic sites should automate generation, use stable partitions, monitor failures, and refresh files from reliable events. The process should handle partial failures without publishing misleading URL sets.

A product catalog may refresh after publishing, removal, or canonical changes. A multilingual site may split files by a stable locale or host boundary.

Partition Sitemaps by Stable Criteria

Partition sitemaps by content type, locale, host, or another stable owner. Stable groups make generation and troubleshooting easier to trace.

Avoid groups that create duplicates or change ownership on every run. Record which system creates each file and which events trigger updates.

Complement Sitemap Discovery With Rendered Crawling

Olostep documents that its web crawling API uses recursive and rendered crawling to complement sitemap discovery. This is a first-party capability statement, not a universal rule about website behavior.

Results depend on scope, permissions, and page behavior. Rendered crawling does not guarantee complete route discovery.

How Do You Compare Sitemap URLs With Crawled URLs?

A sitemap audit compares the declared URL list with URLs found through a permitted crawl. Normalize both sets and classify each match or difference.

Olostep's Maps endpoint provides a programmatic route for structured URL discovery. Its output can support the comparison, subject to the chosen scope and settings.

Build a Declared-Versus-Observed URL Inventory

A declared-versus-observed inventory preserves each URL and the method that found it. Sources may include sitemap files, robots.txt references, internal links, rendered links, and redirects.

Normalize fragments, tracking parameters, protocols, hosts, casing, and trailing slashes with documented rules. Keep the original URL beside the normalized value for auditability.

In this workflow, crawling means discovering and following URLs; scraping means extracting content from retrieved pages. Olostep’s crawling versus scraping article is a related first-party technical guide.

A practical record can contain these fields:

Original URL: The exact URL found at the source.
Normalized URL: The comparison key produced by documented rules.
Discovery source: The sitemap, robots.txt, internal link, rendered link, or redirect.
HTTP status: The latest observed response status.
Canonical target: The preferred URL declared by the page, when available.
Observed time: The timestamp for the discovery or check.

Classify Coverage and Quality States

Coverage states turn URL differences into review queues. Each state needs one definition and one next action.

State	Meaning	Recommended Review
`declared_only`	Listed in a sitemap but absent from the selected crawl	Check crawl scope, access, and internal links
`crawl_only`	Found through crawling but absent from the sitemap	Review canonical and search intent
`both`	Listed in the sitemap and found through crawling	Confirm status and canonical target
`redirected`	Resolves to another URL	Replace or remove the source URL
`blocked`	Access is restricted for the selected crawler	Review robots.txt and access policy
`noncanonical`	Declares a different canonical URL	Prefer the canonical target
`error`	Returns an error or cannot be processed	Check responses, XML, and retrieval logs
`render_discovered`	Appears only after the selected rendered crawl	Review stability, access, and inclusion intent

A state reflects the selected crawl's scope, settings, and observation time. It does not prove that every crawler will see the same result.

Use the Inventory in Production Workflows

A reconciled inventory can support migrations, monitoring, SEO audits, content reviews, research, RAG, enrichment, and competitive analysis. Its use depends on permissions, scope, normalization, and refresh cadence.

Teams can assign work or trigger checks from each state. Olostep describes related uses in its SEO data workflows capability page.

For a migration, declared_only and redirected URLs can form a review queue. For monitoring, error and noncanonical states can trigger focused checks.

Sitemap FAQs

These sitemap FAQs address common questions about need, formats, indexing, exclusions, updates, and fetch errors. Each answer keeps discovery separate from crawling and indexing.

Does Every Website Need a Sitemap?

No. Google says smaller, comprehensively linked sites may need less sitemap support, while new, large or complex, and media-heavy sites may benefit more.

What Is the Difference Between an XML Sitemap and an HTML Sitemap?

An XML sitemap is a supported machine-readable format. An HTML sitemap is an ordinary internal-link page for navigation, not a submitted Google sitemap format.

Will a Sitemap Guarantee That Google Indexes My Pages?

No. A sitemap can support discovery, but it does not guarantee that Google will crawl, index, or rank a listed page.

What Pages Should Be Left Out of a Sitemap?

Leave out redirects, errors, duplicates, noncanonical variants, and pages intentionally excluded from search. Include preferred canonical URLs intended for search results.

How Often Should a Sitemap Be Updated?

Update a sitemap when the preferred URL inventory changes or verified modification data needs an update. Avoid arbitrary refresh dates.

Why Does Search Console Say It Couldn't Fetch My Sitemap?

Search Console may report access, URL, redirect, XML, scope, or server problems. Correct the issue, request the file directly, and review the report again.