Sitemap.xml for AI Crawlers: What to Include and What to Skip

The unsexy foundation

Sitemap.xml looks like a relic. It is not. AI crawlers (GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended) still consult sitemap.xml as a primary discovery mechanism. A clean, complete sitemap means your pages get crawled. A missing or broken sitemap means slow or partial discovery.

This is one of those AEO topics where the right answer has not changed much from 2010 SEO advice. The mechanics are the same; the stakes are higher because uncrawled pages cannot be cited.

What AI crawlers read from sitemap.xml

Three fields drive crawler behavior.

loc (URL). The page URL. Required. Used as the canonical address to crawl.

lastmod. When the page last changed. Used to decide whether to re-crawl. Critical for freshness signals - covered below.

priority and changefreq. Hints about page importance and update cadence. Most major crawlers ignore them as policy. Set them honestly but do not over-optimize - they are not significantly load-bearing.

The fields engines do not heavily use (priority, changefreq) are not worth obsessing over. The fields they do use (loc, lastmod) need to be correct.

What to include

Three rules.

Include all canonical, public pages you want cited. Blog posts, articles, product pages, documentation, comparison pages, glossary entries, FAQ pages. If you want it cited, it goes in.

Exclude redirects, parameter URLs, and noindex pages. Crawlers waste budget on redirected URLs. Parameter URLs (?utm_source=...) bloat the sitemap. Noindex pages should not be in the sitemap because the engine cannot index them anyway.

Exclude staging, internal, and admin pages. Even if they are technically reachable, they should not be in the public sitemap. Use robots.txt to block crawl as well.

The right size: most sites have 100-10,000 sitemap entries. Above 50,000 entries, split into multiple sitemap files indexed by sitemap_index.xml. Above 500,000 entries, consider whether all those URLs really should be cited or whether you have a parameter-bloat problem.

What to exclude

Four categories of common mistake.

Tag and category archive pages with thin content. A category page that lists 5 blog posts adds little citation value. Engines crawl it, find nothing extractable, and waste budget. Either fortify these pages with genuine content or exclude them from the sitemap.

Pagination URLs (page 2, 3, 4...). These rarely cite. Including them spreads crawl budget across low-value URLs. Many CMS templates auto-generate them; manually exclude or use canonical-to-page-1.

Search result pages. Internal site search URLs (/search?q=...) should never be in the sitemap. Crawlers waste budget; some engines penalize sites that publish search-result URLs as canonical.

Auto-generated user pages. User profile pages, author archives with no actual content. Same logic - thin content, wastes crawl budget.

A common audit pattern: sites with 10,000-entry sitemaps that should have 800-entry sitemaps. The engine spends days crawling chaff before reaching the substance. Cleaning the sitemap accelerates citation reach measurably.

lastmod accuracy

The lastmod field is the most-undervalued sitemap signal. Engines use it to decide re-crawl priority. A page with a stale lastmod is treated as stable; a page with a fresh lastmod is re-crawled sooner.

The mistake: some CMS auto-update lastmod on every minor change (header tweak, footer update, comment posted). Engines see constant lastmod churn and either ignore the signal or down-rank for unreliable freshness data.

The right pattern: lastmod moves only when the page content materially changes. A typo fix in the body counts. A deploy that touches a global header does not.

If your CMS does not give you this control, consider generating sitemap.xml from your database with content-aware lastmod logic. Most static-site generators do this correctly by default; many WordPress plugins do not.

Multi-sitemap structures

Large sites split sitemaps. The conventional structure:

/sitemap_index.xml
  /sitemap_blog.xml
  /sitemap_products.xml
  /sitemap_docs.xml
  /sitemap_pages.xml

Each child sitemap covers a content type. The index points to them. Robots.txt references the index.

This structure helps engines crawl in priority order. They can fetch the blog sitemap first, the docs sitemap second, etc. It also keeps individual sitemap files under the 50K entry / 50MB size limits.

For small sites (under 5,000 URLs), a single sitemap.xml is fine. The split helps at scale.

How robots.txt and sitemap.xml interact

The relationship is simple but often misconfigured.

robots.txt should reference the sitemap:

Sitemap: https://example.com/sitemap.xml

Engines that fetch robots.txt then look for the Sitemap: line and follow it. This is the most reliable discovery path.

Sitemap.xml should not contain URLs that robots.txt blocks. The mismatch (sitemap says "crawl this," robots.txt says "do not crawl this") confuses engines and produces unpredictable behavior.

Audit both files together. Make sure they agree on what is and is not crawlable.

How Citevera scores this

The audit checks sitemap.xml presence, structure, content, and accuracy. It flags missing sitemaps, sitemaps with stale lastmod values, sitemaps containing redirected or noindex URLs, and bloated sitemaps with thin-content URLs.

It also checks robots.txt for the Sitemap: directive and verifies that sitemap.xml URLs match the robots.txt allow rules. The crawlability axis weights all of this.

Run a free Citevera audit to check your sitemap setup

Frequently asked questions

Do AI crawlers respect sitemap.xml the same way Google does?

Mostly. The major AI crawlers fetch sitemaps and use them for discovery. Some weight lastmod more heavily than Google does (Perplexity especially, which prizes freshness).

Should I use XML sitemaps or simpler text sitemaps?

XML. Plain-text sitemaps are simpler but lack lastmod and other useful fields. The investment in proper XML pays back in better crawler behavior.

What about news, image, and video sitemaps?

Submit them where they apply. News sitemaps for sites with regular news content. Image and video sitemaps for media-heavy sites. They help engines surface your content in modal-specific answers (image carousels, video summaries).

How often should sitemap.xml be regenerated?

Whenever content changes. Most CMS regenerate on publish. Static-site generators regenerate on build. Manual sitemaps are a recipe for drift; automate generation.

Will adding sitemap.xml to a site that does not have one move citation rate?

For most sites yes, modestly. Engines that previously discovered pages slowly via crawl now find them faster via sitemap. Effect is most noticeable on newer or less-linked pages that were taking time to be discovered.

Should noindex pages be listed in sitemap.xml?

No. Sitemap.xml should only contain canonical, indexable, public URLs. Noindex pages waste crawl budget and create signal mismatches. If you want a page available but not cited, do not include it in sitemap; if you want it cited, remove the noindex.

How big can sitemap.xml be before I need to split?

50,000 URLs or 50MB uncompressed, per Google's guidelines that other engines also follow. Above that, split into multiple sitemap files referenced from a sitemap_index.xml. Most sites do not approach the limit; very large sites often hit it on individual sitemap files even when the total fits.

Do AI crawlers respect the priority and changefreq fields?

Most major engines treat them as advisory and ignore them in practice. The fields are not harmful to include but are not load-bearing either. Spend your effort on lastmod accuracy and URL inclusion rather than priority/changefreq tuning.