Deduplication
Deduplication is the process in which Google groups similar pages and then picks one of them to represent the group in the index. The chosen page is the canonical page, and the others are duplicates. Google also calls this process canonicalization. Thanks to it, search results show only one version of duplicated content. A site owner's choice of canonical page is a hint for Google, not a binding rule.
What is deduplication?
Deduplication is the removal of repetitions from a search engine's index. Google does not show several addresses with the same content in its results, only one, the one it considers most representative.
Gary Illyes of Google explained in the “Search Off the Record” podcast in 2020 that deduplication has two stages. First Google builds clusters of duplicates, then it picks a leading page from each cluster.
| Feature | Description |
|---|---|
| Type | search engine indexing process |
| Stages | grouping duplicates (clustering) and choosing the canonical page |
| Result | one canonical page representing the group |
| Other name | canonicalization |
| Effect on results | Google usually shows only the canonical page |
| Effect on crawling | duplicates are crawled less often than the canonical page |
According to Google's documentation, duplicate content within a site is normal and does not violate spam policies. It can, however, make it harder to track how content performs in search results. Originality and duplication of content are usually checked during an SEO audit.
Deduplication should not be confused with content quality assessment. Pages with thin or duplicated content were rated low by, among others, the Google Panda algorithm.
How does Google group pages and choose the canonical page?
Google first determines the main content of each page, then puts pages with the same or very similar content into a cluster. From the cluster it picks the page that, based on the signals collected, is the most complete and useful.
According to Gary Illyes, Google calculates checksums for each page, a kind of content fingerprint. Before that, it removes repeated elements such as navigation and the footer. What remains is the main content of the page, which Google calls the centerpiece. Pages with matching checksums go into one cluster.
| Stage | What happens | Source |
|---|---|---|
| Determining the main content | Google skips navigation and the footer and analyzes the central content | Gary Illyes, 2020 |
| Clustering | pages with the same or very similar content go into one cluster | Google documentation |
| Choosing the canonical page | Google picks the most complete and useful page from the cluster | Google documentation |
| Indexing and crawling | the canonical page is crawled most often, duplicates less often | Google documentation |
John Mueller of Google summed up the difference in a 2024 podcast. Clustering is gathering the pages Google considers to be the same. Canonicalization is choosing the best page from that group.
Allan Scott of the Search team noted in the same episode that many canonicalization problems are really clustering problems. Two different pages can end up in one cluster even though they should not.
Google treats language versions as duplicates only when the main content is in the same language. In results, Google may show a duplicate instead of the canonical page when it suits the user better, e.g. the mobile version on a phone.
Which signals affect the choice of canonical page?
The choice of canonical page is affected by many signals, including redirects, the rel=canonical attribute, the HTTPS protocol and the presence of the address in the sitemap. According to Allan Scott of Google, there are currently about 40 signals.
Google's documentation lists the methods of indicating the canonical page in order from strongest.
| Method | Signal strength according to Google |
|---|---|
| Redirect | a strong signal that the redirect target should become canonical |
| rel=canonical attribute | a strong signal that the indicated address should become canonical |
| Sitemap | a weak signal supporting addresses in the sitemap |
In 2024 Allan Scott listed example signals: redirects, the rel=canonical attribute, x-default, sitemaps, links and PageRank. In 2020 Gary Illyes spoke of more than 20 signals whose weights Google sets using machine learning.
According to Allan Scott, the rel=canonical attribute works at both stages. First it tries to put two pages into one cluster. Once the pages are in one cluster, it becomes a signal for choosing the canonical page.
Google advises against using the robots.txt file and the noindex directive to indicate the canonical page. Noindex removes the page from the search engine completely, so the recommended solution is the rel=canonical attribute.
AMP pages are an example of an alternate version. According to Search Console Help, the canonical page for an AMP page should usually be the non-AMP version.
How do you check deduplication in Google Search Console?
Deduplication is checked in Google Search Console, in the page indexing report and in the URL Inspection tool. The tool shows which page the owner indicated and which one Google chose.
| Status in the indexing report | Meaning according to Google |
|---|---|
| Duplicate without user-selected canonical | Google considered another page canonical and does not show this one in results |
| Duplicate, Google chose different canonical than user | Google considered another address a better canonical page |
| Alternate page with proper canonical tag | the page correctly points to the indexed canonical page |
According to Search Console Help, the status “Duplicate without user-selected canonical” is not an error. Google deliberately does not show duplicate pages.
The URL Inspection tool shows two values: the user-declared canonical and the Google-selected canonical. When Google's choice is unexpected, Google recommends marking the canonical page explicitly.
Frequently asked questions
Q.Is deduplication a Google penalty?
No. Deduplication is a standard stage of indexing, not a penalty. According to Google's documentation, duplicate content within a site is normal and does not violate spam policies. Google simply shows one version in results instead of several. A duplicate status in Search Console means the process is working as intended.
Q.How is deduplication different from canonicalization?
Google often uses the two names interchangeably. Gary Illyes described deduplication as the whole process, which includes building clusters of duplicates and canonicalization. Canonicalization is therefore the stage of choosing the page that represents the cluster. John Mueller summed up that clustering gathers pages that are the same, and canonicalization picks the best of them.
Q.Does Google always respect the rel=canonical tag?
Not always. According to Google's documentation, indicating a canonical page is a hint, not a rule. Google may choose another page, e.g. when it considers it more complete. If the declared canonical page is not similar to the current page, Google will never choose it as canonical.
Q.How do you check which page Google chose as canonical?
The simplest way is the URL Inspection tool in Google Search Console. It shows the user-declared canonical and the Google-selected canonical. The page indexing report, in turn, groups addresses with duplicate statuses. This lets you find pages where Google chose a different address than the owner.
Q.What sources is this description based on?
The definition of canonicalization, how the canonical page is chosen and the causes of duplicates are described in the Google Search Central documentation “What is URL canonicalization”. The methods of indicating a canonical page and their strength are given in the documentation “How to specify a canonical URL”. Duplicate statuses are described in Search Console Help articles on the page indexing report and the URL Inspection tool. Statements by Google employees from the “Search Off the Record” podcast are quoted by Search Engine Roundtable in articles from 2020 and 2024.
Let's talk growth
Send a brief or drop your contact. We'll reply within 24h.
Get a quote