What is Duplicate content?

Duplicate content is content that appears, word for word or very nearly, at more than one URL. It can happen within one site (the same product page reachable through four different addresses) or across sites (an article republished by partners or scraped by a copycat).

Most duplication is technical and accidental. Every one of these URLs might serve the same page:

http://example.com/shoes/
https://example.com/shoes/
https://www.example.com/shoes
https://www.example.com/shoes/?utm_source=newsletter
https://www.example.com/shoes/?sort=price&color=blue
https://www.example.com/Shoes/

To you, that’s one page. To a crawler, it’s six URLs that must be fetched, compared and grouped.

The penalty myth

There is no general “duplicate content penalty”. Google groups duplicates into a cluster, picks one URL as the canonical, and shows that one in results; the rest are treated as alternates. Google’s spam policies only come into play when duplication is deceptive — scraping other sites or mass-producing near-identical pages to manipulate rankings.

The real costs are subtler:

• Google may pick the wrong canonical, such as a parameter URL or the http version, so the address you promote isn’t the one that ranks.
• Signals get split when backlinks and internal links point to different versions.
• Crawling is wasted on variants instead of new or updated pages.
• AI systems see the same passage several times and may attribute it to whichever copy they encountered first, including a syndication partner.

How to consolidate duplicates

Google’s guide to consolidating duplicate URLs ranks the methods by strength. Use them together so they agree:

1. Redirects — a 301 redirect from http to https and from non-www to www (or the reverse) removes protocol and host duplicates entirely.
2. The canonical tag — every variant declares the preferred URL with a canonical tag:

   <link rel="canonical" href="https://www.example.com/shoes/">

3. Consistent internal links and sitemaps — link only to the canonical form, and list only canonical URLs in your sitemap.

Common mistakes

• Canonical tags pointing to a URL that redirects, returns 404 or is marked noindex.
• Mixing signals: the canonical says one URL, the sitemap lists another, internal links use a third.
• Blocking duplicate URLs in robots.txt. Google then can’t fetch them to see the canonical tag, so the duplicates can remain indexed.
• Reusing the same manufacturer description on hundreds of product pages with nothing original added — not a penalty, but it gives Google no reason to rank your page over every other shop with the same text. That edges into thin content.

How to check

The Canonical Checker finds the canonical in your HTML and HTTP headers, checks that it is absolute, returns 200 and is indexable, and tests whether the http/https and www/non-www versions redirect to one address. For duplicate titles and descriptions across many pages, a site crawl with the Website SEO Audit lists the clusters.

Frequently asked questions