Last updated: September 25, 2026
An xml sitemap is a machine-readable file that lists the URLs on a website you want search engines to know about, together with the date each one last changed. It is a discovery aid. It does not force a crawl, it does not confer priority, and it does not affect ranking. What it does, when it is built well, is give Google a trustworthy candidate list and a trustworthy freshness signal, and on a large site that is the difference between a crawl that finds the pages that matter and one that spends its budget on variants. This guide sets out what Google actually reads from the file and what it ignores, how to split a large site across a sitemap index, what to include and exclude, how the file relates to the “discovered, currently not indexed” queue, and why the sitemap has become the natural scope file for automated on-page optimisation.
Google’s documentation is unusually direct on this topic. Its guide to building and submitting a sitemap states that a single sitemap is limited to 50MB uncompressed or 50,000 URLs, that Google ignores the priority and changefreq values, and that it uses the lastmod value only if it is consistently and verifiably accurate, where a significant update means a change to the main content, structured data or links on the page rather than, for example, an updated copyright date. Everything else in this guide follows from those three facts.
If you want the sitemap kept honest automatically, only canonical indexable URLs with truthful change dates, and used as the scope for your on-page rules, sign up for free and NytroSEO will start with an inventory of what the file currently contains.
The reason the topic needs a 2026 guide is that most sitemap advice is a decade old. Sites still set a priority on every URL, still set changefreq to daily in the hope of faster crawling, and still list every URL the CMS can generate, including redirects, parameter variants and pages marked noindex. Google has said for years which parts of the protocol it reads and which it discards, and Bing published the same position in 2025. The file has one job left, and it does that job well only when it is lean and truthful.
Lee Agam, founder and CEO of NytroSEO, treats the sitemap as a statement of intent that most sites make carelessly. In his experience with large sites, the file is generated by a plugin or a build step nobody has reviewed since launch, so it lists whatever the CMS considers a page, stamps every entry with today’s date on every rebuild, and quietly teaches Google that neither the list nor the dates mean anything. His rule is that the sitemap should describe the site you want crawlers to see, not the site the database contains: canonical URLs that return 200 and are meant to be indexed, with a lastmod that changes only when the content does. He also uses it for a second purpose, as the definition of “pages in scope” for automated metadata and structured-data rules, because a file that already says which URLs matter is the right boundary for automation to respect.
What an xml sitemap does (and does not do) for sitemap and seo
An xml sitemap announces URLs and freshness to search engines. In the sitemaps.org protocol each entry carries a location, loc, and optionally a last-modified date, lastmod, a change frequency, changefreq, and a priority. Google reads loc and, when it trusts them, lastmod. Bing takes the same position. The other two fields are valid in the protocol and inert in practice.
What the file does for sitemap and seo is therefore narrow and real. It gives crawlers a candidate list, which matters most for pages that are weakly linked or newly published. It gives them a freshness signal per URL, which shapes recrawl decisions when the dates are honest. And it gives Search Console a set of submitted URLs against which the Page indexing report can show coverage, which is the only way to see which of the pages you intended to be indexed actually are.
What it does not do matters as much. It does not force a crawl: a URL in the sitemap that no page links to and that Google judges low priority can sit in “discovered, currently not indexed” indefinitely. It does not consolidate duplicates; canonical tags and redirects do that, and our guide to the alternative page with proper canonical tag status covers the interaction. It does not affect ranking. And it does not make a non-indexable page indexable; a noindex page in the sitemap produces a “submitted URL marked noindex” conflict, and Google follows the noindex.
One more thing it no longer does: Google’s sitemap ping endpoint was deprecated, so notifying Google of a change means resubmitting through Search Console or the Search Console API, or simply letting lastmod do its work.
How to create xml sitemap files for large sites: index files and splitting
How to create xml sitemap files for a large site comes down to one structural decision: split by template, and split well before the limits.
The structure above splits by template so that coverage problems are visible per section; the index file is what you submit.
A sitemap index is a file that lists other sitemaps rather than URLs. Submit the index once in Search Console and Bing Webmaster Tools; each child sitemap is discovered from it. Each child may hold up to 50,000 URLs or 50MB uncompressed, whichever binds first; on image and hreflang sitemaps the byte limit usually binds first. A file over either limit may be rejected or only partly read, and Search Console does not warn you before that happens.
Split by template rather than by count. Products in one child, categories in another, articles in a third, location pages in a fourth. The reason is diagnostic: when the Page indexing report shows a coverage problem, a template-split sitemap tells you immediately which section is affected, whereas a numbered split, sitemap-1 to sitemap-40, tells you nothing. Large templates that exceed the limit split further by date or ID range within the template.
Give each child sitemap its own lastmod in the index, set to the most recent change inside it. That lets Google skip children that have not changed and fetch the ones that have.
Generate dynamically from the CMS or a build step rather than maintaining the file by hand, but review what the generator includes. WordPress plugins, static-site builders and ecommerce platforms each have defaults that need checking: some include tag and author archives, some include attachment pages, some set every lastmod to the build time. Our older note on whether to resubmit your sitemap every month still holds: resubmission does nothing if the file has not changed, and a dynamic file makes resubmission unnecessary.
What to include, exclude and mark
The rule for inclusion is short. List a URL if, and only if, it is the canonical version, it returns 200, it is indexable, and you want it indexed.
The sorter above applies the one inclusion rule; everything in the right-hand tray contradicts a signal you have set elsewhere.
Include: canonical product, category, article, guide and location pages that return 200 and carry no noindex. Include the pages that are hard to reach through navigation, because those are the ones discovery helps most.
Exclude: redirects, because they waste crawl on a hop; non-canonical variants, including parameter, tracking, sort and filter URLs, because they contradict your canonical tags; pages marked noindex, because they create a conflict Google resolves against you; 404 and 410 URLs; paginated and faceted duplicates; thin utility pages such as login, cart, search results and tag archives; and anything blocked in robots.txt, which Google cannot fetch regardless.
Mark lastmod, and mark it honestly. Emit it from the content’s real change date, ideally with time and timezone offset, and change it only when the main content, structured data or links change. Do not set it from the build time, the database row’s updated_at that ticks on every view count, or today’s date on every regeneration. If your platform cannot produce an honest date, omit the field; Google says omitting it beats faking it, because a sitemap in which every URL changed today teaches Google to ignore the field for the whole site.
Drop priority and changefreq. They are ignored, they add bytes, and on a 50,000-URL file the bytes are not free. Keep the file valid XML, UTF-8 encoded, with entity-escaped URLs, and reference the index from robots.txt as well as submitting it in Search Console; the two do different jobs, one for any crawler that reads robots.txt, one for Google’s reporting.
Google xml sitemaps guidance also supports specialised sitemaps for images, video and news; use them where those content types matter, and remember the byte limit binds earlier on them.
How do sitemaps affect discovered currently not indexed pages?
How do sitemaps affect discovered currently not indexed pages? They cause many of them and fix few of them on their own.
The status means Google knows a URL exists but has not crawled it. A sitemap is often how Google learned the URL existed, so a large sitemap full of weakly linked pages is a machine for producing the status. The sitemap announced the pages; nothing else about the site told Google they were worth fetching.
The fix is therefore subtraction plus links. Remove from the sitemap the URLs that should never have been discoverable, so the queue Google is choosing from gets smaller and more credible. Keep the lastmod honest, so the pages that did change are the ones Google recrawls. Then link the pages you kept from pages Google already crawls often, because internal links are the strongest lever for a discovered URL and the sitemap is the weakest. Our guide to discovered currently not indexed sets out the full order of operations, and our guide to crawl efficiency covers the triage on very large sites.
Two habits close the loop. Read the Sitemaps report in Search Console after every regeneration; it shows three states, success, has errors and could not fetch, and no warning tier, so an error is the only signal you get. And compare submitted URLs against indexed URLs per child sitemap in the Page indexing report; a template whose indexed count is far below its submitted count is where the crawl is being declined, and that is the template to work on.
An AI search sitemap view: scoping automation by sitemap
An AI search sitemap is not a new file format. It is a way of using the ordinary xml sitemap for two things the AI era makes valuable: knowing which pages you consider important, and applying rules to exactly those pages.
On the first point, honesty is required about what is not documented. No major AI crawler operator, not OpenAI, not Anthropic, not Perplexity, publishes documentation saying its crawler reads XML sitemaps for discovery. Some practitioners report seeing AI crawler requests for sitemap files in their logs; treat that as observed behaviour rather than a documented feature, and do not build a strategy on it. What every AI system does read is the indexable HTML that Google and Bing index, and the sitemap’s job is to keep that index accurate.
On the second point, the sitemap is the best boundary a large site has for automated on-page work. It already answers the question automation has to ask first: which URLs are in scope? A rule that generates titles, descriptions, canonical hints, alt text or structured data should run over the canonical indexable URLs the sitemap lists and nothing else, so that variants, utility pages and retired URLs are never touched. When a page is added to the sitemap it enters scope; when it is removed, it leaves. That is how NytroSEO scopes the metadata layer it applies through its header snippet: by sitemap, with a preview before rules apply, a change log and rollback. It does not build or fix the sitemap itself, which remains platform work, but it depends on the file being honest, which is one more reason to make it so. Our guide to meta-tag automation for large websites shows the scoping in practice, and our note on what large ecommerce sites do differently covers the catalogue case. Book a strategy meeting with the NytroSEO team if you manage a large site or a client portfolio and want the sitemap audit and the scoping set up as one project.
A sitemap hygiene checklist
- Only canonical, indexable URLs that return 200; no redirects, noindex pages, parameters or blocked URLs.
- Split by template into children under 50,000 URLs and 50MB; submit the index; give each child its own lastmod.
- lastmod from real content changes, with time and offset; omit rather than fake.
- No priority, no changefreq.
- Valid UTF-8 XML, entity-escaped; referenced in robots.txt and submitted in Search Console and Bing Webmaster Tools.
- Sitemaps report checked after each regeneration; submitted-vs-indexed compared per child monthly.
- Sitemap used as the scope for automated on-page rules.
Frequently Asked Questions
A single sitemap file can list up to 50,000 URLs and must be no larger than 50MB uncompressed, whichever limit is reached first. Larger sites use a sitemap index that references multiple child sitemaps, typically split by template so that coverage problems are visible per section.
No. Include only canonical, indexable URLs that return 200 and that you want indexed. Leave out redirects, noindex pages, parameter and filter variants, paginated duplicates, utility pages and anything blocked in robots.txt. A lean sitemap helps Google trust the file and prioritise the pages that matter.
Yes, when it is accurate. Google uses lastmod to decide what to recrawl, but only if the values are consistently and verifiably accurate and reflect significant changes to content, structured data or links. Setting every URL to today’s date on each rebuild teaches Google to ignore the field for the whole site; omitting it is better than faking it.
Not alone. A sitemap tells Google a URL exists and when it changed; it does not force a crawl, and a large sitemap of weakly linked pages is often what produces the status. Removing low-value URLs, keeping lastmod honest and adding internal links from well-crawled pages together shorten the wait.
No major AI crawler operator documents reading XML sitemaps for discovery, so treat any reports of it as observed behaviour rather than a feature to rely on. What AI systems demonstrably read is the indexable HTML that Google and Bing index; the sitemap’s value for AI search is keeping that index accurate and providing a clean scope for automated on-page rules.
Ready to make the sitemap say what you mean?
A sitemap that lists everything and dates everything today tells crawlers nothing. One that lists the canonical pages you want indexed, with honest change dates, is both a better discovery file and the right boundary for automation. Sign up for free to have NytroSEO inventory your sitemap and apply on-page rules within it, or book a strategy meeting if you manage a large site or a client portfolio and want the sitemap strategy and scoping designed as one project.
