Meta Robots Guide — Noindex, Nofollow & X-Robots

The meta robots tag and the X-Robots-Tag HTTP header are two implementations of the same directive system. The meta tag works for HTML pages; the HTTP header works for any resource type (PDFs, images, JavaScript files). Both use the same directive keywords and are processed identically by Googlebot.

Key Takeaways

noindex prevents the page from appearing in search results; nofollow tells crawlers not to follow outbound links on the page — they are completely independent directives
Blocking a page with robots.txt does NOT prevent indexing — Google can still index a page it cannot crawl (via anchor text from other pages)
X-Robots-Tag is the only way to apply robots directives to non-HTML resources like PDFs, images, or feeds
Googlebot respects the most restrictive directive when conflicting instructions exist between robots.txt and meta robots
The most common mistake: a staging disallow left in robots.txt on production, or a noindex accidentally applied site-wide via a CMS setting

The Full Directive Reference

Directive	Effect
`index`	Default — page may appear in search results (can omit this)
`noindex`	Do not include this page in search results
`follow`	Default — crawl and pass PageRank through outbound links
`nofollow`	Do not crawl or pass PageRank through outbound links on this page
`nosnippet`	Do not show a text snippet or video preview in search results
`noimageindex`	Do not index images on this page
`noarchive`	Do not show a cached version of this page
`none`	Equivalent to `noindex, nofollow` combined
`all`	Equivalent to `index, follow` (the default — rarely needed explicitly)

Meta Tag vs X-Robots-Tag — When to Use Each

For HTML pages, both work. The meta tag is simpler and more commonly implemented:

<meta name="robots" content="noindex, nofollow">

For non-HTML resources — PDFs, images, JSON feeds, CSV exports — only the HTTP header can carry the directive:

X-Robots-Tag: noindex, nofollow

You can also target specific crawlers with either method:

<meta name="googlebot" content="noindex">
X-Robots-Tag: googlebot: noindex

Noindex vs Robots.txt Disallow — The Critical Difference

This is the most commonly misunderstood distinction in crawl control:

Method	Prevents crawling?	Prevents indexing?
`robots.txt Disallow`	Yes	No — Google can still index a blocked page via external links
`meta noindex`	No — Googlebot still fetches the page	Yes — page removed from search results
Both combined	Yes	Uncertain — Google can't read the noindex if blocked from crawling

Don't block pages you want to noindex with robots.txt. If Googlebot can't fetch the page, it can't read the noindex tag. A disallowed page with a noindex meta tag is effectively just disallowed — Google may still show the URL in results without a snippet if other sites link to it.

When to Use Noindex

Pages that should typically carry noindex:

Thin or duplicate content — pagination pages beyond page 2, filtered product views, date-archive URLs on WordPress
Account and utility pages — login, register, password reset, cart, checkout — no SEO value, shouldn't appear in search
Thank-you and confirmation pages — post-conversion pages that exist in funnel flow, not as search entry points
Internal search result pages — infinite combinations of query parameters that could produce duplicate content at scale
Staging/preview environments — if they're publicly accessible, add noindex at server level (X-Robots-Tag) rather than relying on robots.txt

Use the SEO Site Audit to identify which pages on your site are currently flagged as noindex — and verify that the noindex list matches what you intend. CMS plugins can accidentally apply noindex site-wide.

When to Use Nofollow on the Page

Page-level nofollow (in the meta robots tag) is rarely the right tool — it's a blunt instrument that stops all outbound link equity passing from the page, including to your own site's internal pages. Use it only when a page genuinely has no outbound links worth following (e.g. a print-optimised version).

For selectively nofollowing individual outbound links, use the link-level rel="nofollow" attribute on the <a> tag instead.

Diagnosing Accidental Noindex

The most impactful mistake in robots management is accidentally noindexing pages that should be indexed. Signs of this include: ranking drops across multiple pages at once, Google Search Console showing a spike in "Excluded — noindex" pages in the Coverage report, or organic traffic dropping to zero after a site change.

Common causes:

WordPress "Discourage search engines" checkbox left on after migrating from staging to production
A CMS plugin applying noindex to all pages while its setting is toggled on
A theme or page builder injecting a noindex tag on all pages as a default
A developer adding <meta name="robots" content="none"> as a quick way to hide content temporarily, never reversed

Find Noindexed Pages on Your Site With One Crawl

Free, no signup. The SEO Site Audit crawls every page and flags noindex in the "Indexable" column — instantly showing you which pages are excluded and whether that matches your intent.

Try the SEO Site Audit Free →

Sunny Pal Singh

Fellow · Technical Director — AI Infrastructure, Cloud Orchestration & Network Automation

Sunny is a Fellow and Technical Director specialising in AI infrastructure, cloud orchestration, and network automation. With hands-on depth across AWS, Azure, GCP, Red Hat OpenStack, and OpenShift, he leads high-performing teams of architects and engineers building transformative solutions at scale. He built ByteWaveNetwork to bring the same engineering rigour to everyday web tooling.

Affiliate disclosure: Some links on this page may be affiliate links. We only mention tools we've personally used and have an honest opinion about. Affiliate revenue helps keep ByteWaveNetwork's tools free and maintained. We are not paid by any of the tools compared in this article for favorable coverage.