Sitemap

Discovery for the pages nothing links to.

What it is

An XML sitemap lists your canonical URLs and when they last changed. A sitemap index points at several sitemaps, which is how large sites stay under the per-file limits.

Referencing it from robots.txt with a Sitemap: line is what makes it discoverable without anyone guessing the path.

Why an agent cares

Anything reachable only through a JavaScript-driven menu, a search box, or pagination is effectively invisible without a sitemap.

It is also the cheapest freshness signal you have — lastmod tells a crawler what to re-fetch rather than making it re-crawl everything.

How to fix it

Generate it at build time so it can never drift from the routes that actually exist.

1. 01Next.js: export a sitemap.ts from the app directory.
2. 02Include only canonical, indexable URLs — no redirects, no noindex pages.
3. 03Add Sitemap: https://example.com/sitemap.xml to robots.txt.

How we test it

We follow the Sitemap: lines in robots.txt first, then fall back to /sitemap.xml and /sitemap_index.xml, and confirm a real urlset or sitemapindex document comes back.

Related