Google Search Console's Coverage report surfaces crawl errors after they've already happened — sometimes weeks after a deployment introduced them. A proactive crawl with a dedicated link checker catches the same issues before they compound. This guide covers every major crawl error type, what causes each one, and exactly what to do to fix it.
Key Takeaways
- Crawl errors divide into site errors (affect the whole domain — DNS, server, robots) and URL errors (affect individual pages — 404, soft 404, 403)
- 404 errors waste crawl budget; broken internal links are the primary source — fix links before adding redirects
- Soft 404s are the hardest to detect: they return 200 but show no useful content — Google's Coverage report flags them explicitly
- Redirect chains beyond 2 hops should be collapsed to a single 301 pointing directly at the final destination
- 5xx server errors must be resolved quickly — Googlebot will retry, but prolonged 5xx on important pages triggers deindexing
What Are Crawl Errors?
A crawl error occurs when Googlebot requests a URL and cannot get a valid, usable response. There are two categories:
Site errors affect the entire domain and prevent Googlebot from crawling any part of the site. These include DNS resolution failures (the domain doesn't resolve), server connection errors (connection refused or timed out), and robots.txt fetch failures (the robots.txt file itself returns an error). Site-level errors appear in Google Search Console under Settings → Crawl Stats.
URL errors affect individual pages. These are the errors most sites deal with day-to-day: 404 not found, soft 404, access denied (403), redirect chains, server errors (5xx), and unauthorised (401). URL errors appear in the Coverage report.
Types of Crawl Errors and What Causes Them
| Error type | Cause | SEO impact | Fix |
|---|---|---|---|
| 404 Not Found | Page deleted, URL changed, broken internal link | Wastes crawl budget; deindexed after repeated crawls | Fix the broken internal link; add 301 if URL changed |
| Soft 404 | Empty template returns 200 (deleted product, no search results) | Crawl budget waste; dilutes site quality signals | Return 404/410, add real content, or noindex the page |
| DNS error | Domain doesn't resolve, DNS propagation incomplete | Googlebot cannot crawl anything on the domain | Verify DNS records; check propagation with DNS Lookup tool |
| Server error (5xx) | Application crash, overloaded server, misconfigured proxy | Retry crawls; deindexing after prolonged errors | Fix the application error; use 503 + Retry-After during maintenance |
| Redirect error | Too many hops (chain), redirect loop (A→B→A) | Googlebot abandons after ~5 hops; equity diluted at each hop | Collapse chains to single 301; fix loops with Redirect Tracer |
| Access denied (403) | IP blocks, login required, bot detection triggering on Googlebot | Page treated as not crawlable; may deindex over time | Verify Googlebot isn't being blocked by WAF or login middleware |
How to Find Crawl Errors
Google Search Console — Coverage Report
The Coverage report (Index → Coverage) shows all URLs Google has attempted to crawl, organised by status: Valid, Valid with warnings, Excluded, and Error. The Error tab lists pages that Google tried to crawl and couldn't index due to errors. The Excluded tab includes soft 404s under "Crawled — currently not indexed." This is authoritative but always reflects the past — GSC may lag by days to weeks.
Server access logs
Filter access logs for your Googlebot user-agent string: Googlebot/2.1. Every URL Googlebot requested appears here with the response code your server returned. This is more granular than GSC and updates in real time — useful for catching new errors immediately after a deployment.
ByteWaveNetwork Link Checker
Crawls your entire site following all internal links and reports the HTTP status of every URL found. Run it after any deployment or major content change. Every 404 in the output is a broken internal link contributing to wasted crawl budget. The Link Checker also flags redirect chains so you can see which internal links need to be updated to point directly at the final destination URL.
Fixing 404 Errors
The most common crawl error. Fix strategy depends on the cause:
The URL was changed (restructure, migration) — add a 301 redirect from the old URL to the new one. Update all internal links to point directly to the new URL — don't rely indefinitely on redirects for internal links.
The content was deleted — if you don't intend to replace it, return 404 to signal the page is gone. If the deletion is permanent, return 410 (Gone) — Google deindexes 410s significantly faster.
The URL was never correct (typo in an internal link) — fix the internal link. No redirect needed if the URL never existed as a published URL.
Fixing Soft 404s
Soft 404s are pages that return HTTP 200 but contain little or no useful content — a product template after the SKU was deleted, a "no results found" search page, an author archive for an author with no posts. They're harder to detect than real 404s because they don't surface in most monitoring tools.
Three ways to fix them, depending on the situation:
- Add real content — if the page should exist but is thin, populate it. An empty category page becomes useful once it has at least a few products or a description of the category.
- Return the correct status code — if the underlying resource genuinely doesn't exist (deleted SKU, removed profile), return 404 or 410 rather than rendering an empty template.
- Noindex + canonical — if the page must exist at the URL (e.g. for UX) but shouldn't be indexed, add
<meta name="robots" content="noindex">and a canonical pointing to the relevant parent page. Note: noindex still uses crawl budget — the page gets crawled to read the tag.
Fixing Redirect Errors
Redirect chains occur when a URL redirects to a second URL that redirects to a third, and so on. Googlebot follows chains up to roughly 5 hops before giving up. Each hop also dilutes the link equity passing through the chain. The fix: update all internal links to point directly at the final destination URL, and collapse intermediate redirects.
Redirect loops occur when URL A redirects to URL B which redirects back to URL A (or through a longer cycle back to the start). Googlebot abandons after detecting the loop. These often appear after CMS migrations, incorrect rewrite rules, or misconfigured CDN routing. Use the Redirect Tracer to trace any URL's full chain and identify the loop point.
DNS and Server Errors
DNS errors mean the domain doesn't resolve at all — Googlebot cannot reach any page on the site. These typically occur during DNS migrations, domain expiry, or misconfigured records. Use a DNS Lookup tool to verify that your domain resolves correctly from external resolvers. Google's DNS may cache old records for up to 24 hours after changes.
Server errors (5xx) are common during deployments and traffic spikes. A short burst of 5xx errors during a deploy has minimal SEO impact — Googlebot will retry and will find the pages healthy on the next attempt.
503 Service Unavailable with a Retry-After header set to your expected downtime length. This tells Googlebot exactly when to check back and preserves index status. Without Retry-After, Googlebot has no guidance on retry timing and may increase crawl frequency, adding to server load.
How Often Should You Audit for Crawl Errors?
After any of these events, run a fresh crawl audit:
- Deployments — new code can introduce broken links or change URL patterns
- CMS or platform migrations — URL structures often change; redirect maps are never 100% complete
- Content restructures — moving categories, renaming products, archiving old posts
- Domain changes — every internal link and external link pointing to the old domain
For ongoing monitoring, check the GSC Coverage report weekly. Spikes in the "Error" or "Excluded" counts are the fastest signal that something broke.
Find Every Crawl Error on Your Site
Free, no signup. The Link Checker crawls all internal pages and reports 404s, redirect chains, and broken links — the same errors that waste your crawl budget and slow down indexing.
Try the Link Checker Free →Affiliate disclosure: Some links on this page may be affiliate links. We only mention tools we've personally used and have an honest opinion about. Affiliate revenue helps keep ByteWaveNetwork's tools free and maintained. We are not paid by any of the tools compared in this article for favorable coverage.