Key Takeaways
- Start with
site:yourdomain.comin Google to confirm whether your domain is indexed at all — if zero results appear, the issue is site-wide - A
noindexmeta tag is the single most common cause of specific pages not appearing in search results — including pages accidentally noindexed by CMS settings - A
Disallow: /in robots.txt blocks Googlebot from crawling the entire site — any page that's been blocked since launch will have zero index presence - New sites can take 1–4 weeks for initial indexing; new pages on established sites typically appear within days
- Google Search Console's URL Inspection tool is the most direct way to check indexing status for any specific URL
First — Check If You're Indexed at All
In Google, search: site:yourdomain.com
If Google shows results: your domain is indexed, but specific pages may not be. Move to causes 3–9.
If Google shows zero results: the site isn't indexed at all. The problem is likely cause 1 or cause 2.
If Google shows fewer results than expected: some pages are being blocked or excluded. Multiple causes could be at play simultaneously.
Cause 1 — The Site Is Too New
New sites take 1–4 weeks to be discovered and indexed by Googlebot. Googlebot must first discover the site (via an inbound backlink from an already-indexed site, or a submitted sitemap), then crawl it, then decide to index it — each step takes time and happens asynchronously.
Fix: Submit your sitemap.xml in Google Search Console. Add the site to GSC and verify ownership. Once verified, use the URL Inspection tool to request indexing for your most important pages directly.
Cause 2 — Robots.txt Is Blocking Crawling
Disallow: / in robots.txt tells Googlebot not to crawl any page on the site. The most common way this happens in practice: a staging environment robots.txt (which correctly blocks everything) gets deployed to production alongside a code release.
Check: Visit https://yourdomain.com/robots.txt in a browser. If you see Disallow: / under User-agent: * or User-agent: Googlebot, that's the problem. You can also run the Robots.txt Tester to parse the file and test specific paths in one step.
Fix: Remove or correct the disallow rule, then redeploy. Note: after the block is lifted, Google may take another 1–4 weeks to re-crawl and index pages that were previously blocked.
Cause 3 — Noindex Tag on the Page
A <meta name="robots" content="noindex"> tag in the page's <head> explicitly instructs Google not to index that specific page. This is the most common cause for individual pages not appearing in search results, and it's frequently accidental — WordPress and other CMS platforms have a "Search engine visibility" or "Discourage search engines" toggle that adds this tag to every page when enabled.
Check: View source on the affected page (Ctrl+U in most browsers) and search for noindex. Or run the SEO Analyzer across your whole site — it flags every page carrying a noindex tag with a NOINDEX_DETECTED warning so you can see all affected pages at once.
Fix: Remove the noindex tag from pages you want indexed. In WordPress: Settings → Reading → uncheck "Discourage search engines from indexing this site." In other CMS platforms, look for an equivalent SEO or visibility setting.
Cause 4 — Canonical Tag Pointing Elsewhere
If a page contains <link rel="canonical" href="https://other-url.com/page">, Google treats the canonical target as the authoritative version and indexes that URL instead. The page itself won't appear in results — its PageRank and content signals are consolidated to the canonical destination.
Check: View source on the page and search for rel="canonical". The URL in the href attribute is the page Google considers the definitive version.
Fix: If the canonical is incorrect — for example, pointing to an old domain, a staging URL, or the wrong page variant — update it to point to the correct URL. If you intentionally canonicalized to a different page (for example, to consolidate duplicate content), the original page not appearing in results is expected and correct behaviour.
Cause 5 — No Internal Links (Orphan Pages)
Orphan pages — pages with no internal links pointing to them from anywhere else on the site — may never be discovered by Googlebot even if they aren't blocked. Googlebot follows links from page to page to discover content; without any path of links leading to a page, it has no way to find it unless the URL is in your sitemap.
Check: Run the Link Checker on your site. Any page that appears in no other page's "Found On" column is an orphan candidate. Also check whether the page's URL is included in your sitemap — a sitemap entry compensates for missing internal links.
Fix: Add internal links from relevant existing pages to the orphan page. Contextual links in body copy are more valuable than footer links. Also verify the URL appears in your sitemap and submit the sitemap in GSC if you haven't already.
Cause 6 — Page Is Behind a Login or Paywall
Pages that require authentication to access can't be crawled by Googlebot. The crawler will reach the login form, stop, and never see the page behind it. This affects member-only content, account dashboards, admin areas, and any page that returns a redirect to a login screen for unauthenticated visitors.
Fix: For pages you want indexed, ensure they're accessible without authentication. For genuinely gated content such as paid articles, consider using structured data for paywalled content (isAccessibleForFree: false in your Article JSON-LD schema) — this signals to Google that the content exists and is behind a paywall, which allows it to appear in results with a metered indicator.
Cause 7 — URL Parameters Creating Duplicate Content
Sites that generate large numbers of URL variants through query parameters — ?sort=price, ?page=2, ?ref=newsletter — dilute crawl budget and can cause Google to select a canonical URL that isn't the one you intended to appear in search results. Google consolidates near-duplicate URLs and picks one to index; it may not pick the one you want.
Fix: Add <link rel="canonical"> tags on all parameter variants pointing to the clean base URL. Block purely tracking or session parameters in robots.txt or via the URL Parameters tool in Google Search Console. For pagination, use rel="next" / rel="prev" if you need to signal a paginated series.
Cause 8 — Manual Action Penalty
Google can apply a manual action to a site for policy violations — spam, thin content, unnatural inbound links, cloaking, or hidden text. Pages affected by a manual action are removed from or significantly demoted in search results. Unlike algorithmic penalties, manual actions are applied by a Google reviewer and are clearly flagged in Search Console.
Check: In Google Search Console, navigate to Security & Manual Actions → Manual Actions. If a manual action has been applied, it will appear here with a description of the violation type and which pages are affected.
Fix: Address the specific policy violation described in the manual action, then submit a reconsideration request through GSC. Response times vary but typically take 1–4 weeks.
Cause 9 — Server Errors During Crawl
If Googlebot receives 5xx server errors when attempting to crawl a page, it won't index it. Pages that return 500 errors consistently are eventually removed from the index. This is more common than it might seem — pages that work fine under normal traffic may time out or error when Googlebot crawls them, particularly on shared hosting or during high-traffic periods.
Check: Google Search Console → Indexing → Pages → "Server error (5xx)" lists all URLs returning server errors on crawl. Also run the Link Checker to surface 5xx responses across your site from a browser perspective — if pages error for the crawler, they'll likely error for the link checker too.
Fix: Diagnose and fix the underlying server errors. Monitor the GSC Coverage report after deployments for spikes in 4xx and 5xx errors. Set up uptime monitoring if you don't already have it.
Diagnostic Order
For fastest diagnosis, work through the causes in this order — each step narrows the problem before you invest time in deeper investigation:
site:yourdomain.comin Google — is the domain indexed at all, or is this a specific-page problem?- Check
yourdomain.com/robots.txt— is the entire site blocked from crawling? - Run the SEO Analyzer — identifies noindex pages, canonical mismatches, and missing meta tags across the whole site in one pass
- Google Search Console URL Inspection — check the exact indexing status of a specific URL directly, including the last time Googlebot crawled it
- View source for the canonical tag — is the page pointing Google to a different authoritative URL?
- Run the Link Checker — surface orphan pages that have no internal links pointing to them
- GSC Manual Actions — has a manual penalty been applied?
Find Noindex Pages and Orphan Links Across Your Site
Free, no signup. The SEO Analyzer crawls your site and flags every noindex page, canonical mismatch, and missing meta tag — the fastest way to diagnose why pages aren't appearing in Google.
Try the SEO Analyzer Free →Affiliate disclosure: Some links on this page may be affiliate links. We only mention tools we've personally used and have an honest opinion about. Affiliate revenue helps keep ByteWaveNetwork's tools free and maintained. We are not paid by any of the tools compared in this article for favorable coverage.