Key Takeaways
- Noindex removes a page from Google's index entirely — it won't appear in search results. This is different from robots.txt disallow, which blocks crawling but leaves the page indexable
- The meta robots tag goes in
<head>for HTML pages; X-Robots-Tag is for non-HTML resources (PDFs, images) or for applying rules via HTTP header across page types - Noindex + follow is the correct combination for pages with valuable outbound links you don't want indexed (e.g. tag archives, pagination); noindex + nofollow cuts both indexing and link equity flow
- A page can be both noindexed AND have a canonical tag — but a noindex canonical creates a conflict: you're telling Google not to index the page while also declaring it the canonical version. Remove the canonical if noindexing
- Common noindex accident: a staging site with
<meta name="robots" content="noindex">gets pushed to production without removing the tag — silently deindexing the entire site
Two Ways to Implement Noindex
Method 1 — Meta Robots Tag (HTML Pages)
The meta robots tag is the standard way to noindex an HTML page. Place it inside the <head> element — a meta tag in <body> is ignored by Google:
<head>
<meta name="robots" content="noindex">
<!-- or, to also block link equity flow: -->
<meta name="robots" content="noindex, nofollow">
<!-- or, per-bot: -->
<meta name="googlebot" content="noindex">
</head>Using name="robots" applies the directive to all crawlers that respect the standard. Using name="googlebot" targets Google specifically — useful when you want other search engines to index the page but not Google.
Method 2 — X-Robots-Tag HTTP Header
The X-Robots-Tag response header achieves the same result without modifying HTML:
X-Robots-Tag: noindexUse this for:
- PDF files, images, and other non-HTML resources that have no
<head>element - Applying noindex rules to large groups of URLs via server or CDN configuration
- When you can't modify the HTML (e.g. third-party generated pages or legacy CMS templates)
Both methods are equally valid for HTML pages. Pick one per page — do not send both. If both are present and they conflict, Google reads both and the more restrictive directive wins.
When to Use Noindex
Good candidates for noindex are pages that exist for functional reasons but have no standalone search value:
| Page type | Why noindex |
|---|---|
| Admin / login pages | No SEO value; should never appear in search |
| Thank you / confirmation pages | Post-conversion; no search intent |
| Pagination beyond page 2–3 | Thin content; index-dilution risk |
| Tag and category archives (thin) | Duplicate content with actual posts |
| Internal search results | Parameter URLs with no unique content |
| Print-version URLs | Exact duplicates of main pages |
| Staging / dev environments | Content not ready for indexing |
| Low-quality pages not worth improving | Better removed than indexed |
Noindex vs. Canonical — When to Use Which
Both noindex and canonical tags manage duplicate or undesirable content in Google's index, but they solve different problems.
Use canonical when: Two URLs have similar or identical content and one is the preferred version. Example: example.com/product?color=red vs. example.com/product — add a canonical on the parameter URL pointing to the main product page. The parameter URL remains crawlable and indexable via the canonical redirect; you're consolidating signals, not removing them.
Use noindex when: The page genuinely shouldn't appear in search results at all — it's not a duplicate content problem, it's a "this shouldn't be indexed" problem. Admin pages, login pages, and internal search results are noindex candidates, not canonical candidates.
The conflict: If a page has both a <link rel="canonical" href="[self-URL]"> and <meta name="robots" content="noindex">, you're sending contradictory signals — declaring the page the canonical version while also telling Google not to index it. Google usually respects the noindex in this case, but it's a confusing signal. Remove the canonical from noindexed pages to keep the directive clean.
Noindex vs. Robots.txt Disallow
These two directives are frequently confused because they both "hide" pages — but they work at different stages of Google's pipeline.
robots.txt disallow: Prevents Googlebot from crawling the URL. The page may still appear in the index as a "known URL" from external links — just without any content snippet. Disallow does not remove an already-indexed page; it only prevents future crawling.
noindex: Googlebot can crawl the page (to see the noindex directive), but won't include it in search results. This is the only reliable way to remove an already-indexed page from Google's index.
For admin pages and private areas where you want both protection and deindexing: use both — robots.txt disallow to prevent crawling, and noindex as a belt-and-suspenders backup in case the robots.txt rule is misconfigured or bypassed. Just ensure the noindex is in place before adding the disallow, so Google has a chance to process the noindex first.
Common Noindex Mistakes
Staging site pushed to production with noindex. The most catastrophic mistake. A staging environment typically has <meta name="robots" content="noindex"> sitewide. When that configuration deploys to production, the entire site is silently deindexed — often not discovered until weeks later when GSC shows a cliff-drop in indexed pages. Always audit noindex status immediately after any site migration or deployment.
Noindex on pages with canonical. Conflicting signals confuse Google. Remove the canonical from noindexed pages — a noindexed page shouldn't be declared the canonical version of anything.
Noindex on pages with valuable backlinks. External links to a noindexed page still flow PageRank to the page, but the page can't rank. Depending on link equity flow matters here: noindex + follow passes link equity to linked pages; noindex + nofollow cuts the flow entirely. Use noindex + nofollow only on pages where you're certain the link equity loss is acceptable.
Noindex inside <body>. Meta robots in the body is ignored by Google. It must appear inside <head>. CMS themes that inject meta tags in the wrong location are a surprisingly common source of this problem.
Noindex via robots.txt. Google removed support for a non-standard noindex directive in robots.txt in September 2019. Any noindex directive in a robots.txt file is now silently ignored. Use <meta name="robots" content="noindex"> in the page's <head> instead.
Auditing Noindexed Pages
Periodic audits catch accidental noindex before it silently degrades your rankings.
GSC Coverage report — navigate to Coverage → Excluded → "Excluded by 'noindex' tag" to see all URLs currently noindexed and not in the index. Review this list after every major deployment to catch unintended noindex directives.
URL Inspection tool — for individual pages, the URL Inspection tool in GSC shows the exact robots directives Google sees, including whether the page is indexed or excluded, and why. Use this for spot-checking after template changes.
Crawl-based audit — a full site crawl flags every page with a noindex directive, letting you review the complete list in context. The SEO Analyzer crawls your entire site, surfaces every noindexed page in a dedicated filter tab, and makes it easy to spot pages that shouldn't be noindexed. Run it after any CMS update, template change, or deployment.
Find All Noindexed Pages on Your Site
Free, no signup. The SEO Analyzer crawls your entire site and flags every page with a noindex directive — surfacing accidental noindex tags before they silently remove important pages from Google's index.
Try the SEO Analyzer Free →Affiliate disclosure: Some links on this page may be affiliate links. We only mention tools we've personally used and have an honest opinion about. Affiliate revenue helps keep ByteWaveNetwork's tools free and maintained. We are not paid by any of the tools compared in this article for favorable coverage.