Robots.txt Tester — parse and validate crawl rules
Fetch and parse the robots.txt file for any domain instantly. See the raw file content, Disallow and Allow rules grouped by user-agent (Googlebot, Bingbot, and all crawlers via *), declared XML sitemaps, and whether the file is accessible at all. Use it to diagnose crawl blocks, verify robots.txt after site migrations, and confirm that Googlebot can reach your key pages. Free, no signup required.
What is robots.txt and why does it matter for SEO?
Robots.txt is a plain text file at the root of your domain (/robots.txt) that tells web crawlers — Googlebot, Bingbot, and other bots — which pages or directories they are and aren't allowed to crawl. It uses a simple "User-agent / Disallow / Allow" syntax. A misconfigured robots.txt can accidentally block your entire site from being indexed.
Common robots.txt mistakes include: blocking the root path ("/") which prevents indexing of all pages, blocking CSS and JavaScript files (Google needs these to render pages), and forgetting to update robots.txt after a site migration. The Robots.txt Tester fetches your file and parses every rule so you can spot problems before they affect your search rankings.
Frequently asked questions
- Robots.txt controls crawling, not indexing. A Disallow directive tells Googlebot not to crawl a URL, but Google can still index it if other pages link to it — it just won't see the page content. To prevent a page from appearing in search results, use a tag on the page itself, or the X-Robots-Tag HTTP header. Robots.txt is not a reliable privacy mechanism.
- Disallow in robots.txt stops the crawler from visiting a URL. Noindex (via meta tag or X-Robots-Tag header) tells the crawler it can visit the URL but must not include it in the search index. A page blocked by robots.txt can still appear in Google search results if it has inbound links — it will just show without a title or snippet. Use noindex for pages you want out of search results.
- If robots.txt returns a 404 (not found), web crawlers treat it as if there are no restrictions — all pages are crawlable. This is the safest default. You do not need a robots.txt file unless you want to restrict crawlers from specific paths. However, it is good practice to include one that at least declares your XML sitemap location.
- Add a Sitemap directive at the end of robots.txt pointing to each of your XML sitemap URLs. Example: "Sitemap: https://www.example.com/sitemap.xml". If you have a sitemap index file, point to that. Most crawlers (Google, Bing, Yandex) read Sitemap directives and use them to discover content, supplementing their normal crawling. This is especially useful for large sites with many URLs.
Related tools
- → Sitemap Validator — validate every URL in your XML sitemap
- → SEO Site Audit — crawl and score your entire site for SEO issues
- → Link Checker — find broken links, redirects, and canonical mismatches
Related reading: How to Read robots.txt — Rules, Mistakes & Tester