Sitemap Generator That Lists Only the Pages Worth Indexing

An XML sitemap is a list of the URLs you want search engines to know about. A good one holds no surprises: every entry returns 200, is the canonical version of the page and is allowed to be indexed. A sitemap padded with redirects, duplicates and noindex pages sends mixed signals and buries the URLs that matter.

This sitemap generator builds that clean list for you. It starts at the URL you enter, follows internal links the way a crawler does, respects your robots.txt, skips pages marked noindex or canonicalized to another URL, and produces a standard sitemap.xml file you can download and upload to your site.

How to generate an XML sitemap

1. Enter your home page

   Type the address you want crawled, such as https://www.example.com/. We start there and follow your internal links.

2. Let the crawl finish

   We visit up to 300 pages, obey robots.txt and drop URLs that are noindex or point their canonical to another page.

3. Download and submit

   Save sitemap.xml, upload it to your site root, reference it in robots.txt and submit it in Google Search Console and Bing Webmaster Tools.

What belongs in a sitemap (and what doesn’t)

Google treats a sitemap as a hint about which URLs you consider important, not as a command (Google Search Central). Make the hint trustworthy by listing only URLs that are:

• Final: they answer 200 directly instead of redirecting or returning 404.
• Canonical: the exact version named in the page’s own canonical tag, with the same protocol, host and trailing-slash style.
• Indexable: no noindex and not blocked by robots.txt.

Leave out internal search results, sorted or filtered variants (?sort=price), tracking parameters, cart and account pages, and thin tag archives. Google ignores <priority> and <changefreq>, and only uses <lastmod> when it is consistently accurate (Build and submit a sitemap). A lean file with honest dates beats a long one with every field filled.

The sitemap.xml format in one minute

A sitemap is a UTF-8 XML file that follows the sitemaps.org protocol. The minimum is one <loc> per page holding a full, absolute URL:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/</loc>
    <lastmod>2026-10-28</lastmod>
  </url>
  <url>
    <loc>https://www.example.com/services/web-design/</loc>
  </url>
</urlset>

Special characters in URLs must be entity-escaped (an ampersand becomes &). One file may hold up to 50,000 URLs or 50 MB uncompressed; larger sites split the list and reference the parts from a sitemap index. Gzipping the file is allowed, and the limits apply to the uncompressed size.

Uploading and submitting your sitemap

1. Upload the file so it loads at https://www.example.com/sitemap.xml. A sitemap is generally trusted only for URLs at or below its own folder, so the root is the safe choice.
2. Reference it in robots.txt, where every crawler that reads the file can find it.
3. Submit it in Google Search Console (Indexing → Sitemaps) and in Bing Webmaster Tools, then check the status and discovered-URL count a few days later.

User-agent: *
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

Google retired its anonymous sitemap “ping” URL in 2023, so Search Console and the robots.txt line are what count now. For Bing and the other engines that support it, the IndexNow submitter announces new or changed URLs immediately.

Generated file or CMS sitemap?

A generated sitemap is ideal for hand-coded and static sites, small sites without an SEO plugin, or as a reference to compare against the sitemap your CMS already publishes. It is a snapshot, though: a page you publish tomorrow isn’t in the file until you generate and upload it again.

If your platform builds sitemaps automatically, keep using that. WordPress has shipped a core sitemap at /wp-sitemap.xml since version 5.5 (Yoast and Rank Math replace it with their own), and Shopify and Wix publish /sitemap.xml for you. Running our crawl next to it is still useful: a URL in your CMS sitemap that the crawl never reached is often an orphan page with no internal links, or a noindex page that shouldn’t be listed. Check any sitemap’s health with the sitemap validator.

Tips for a sitemap search engines trust

• Fix internal links first: if the crawl can’t find an important page, visitors and search engines probably can’t either.
• Use one URL style everywhere: https, the same host (www or not) and the same trailing-slash rule in links, canonicals and the sitemap.
• Only change lastmod when the main content changes; bumping every date on every deploy teaches Google to ignore the field.
• Regenerate and re-upload the file after launching a section, renaming URLs or deleting pages.
• Open the Sitemaps report in Search Console a few days after submitting and fix any “Couldn’t fetch” or parsing errors.

Limits to keep in mind

The generator reads the HTML your server sends and doesn’t run JavaScript, so links that only appear after scripts load aren’t followed. It stops at 300 pages and can only find pages that something crawlable links to. The file is a snapshot that doesn’t update itself; for large or fast-changing sites, use your CMS’s dynamic sitemap or the whole-site crawl in the website SEO audit.

Last updated Oct 30, 2026 by the Yes SEO team.