Free SEO Tool · by overrank
Free Robots.txt Validator & Checker
Paste your robots.txt — or just enter a domain and we'll fetch it for you. We parse every line, flag syntax errors, list every blocked path per crawler, and call out the common footguns that quietly kill SEO (blocking the entire site, blocking your sitemap, allow/disallow conflicts).
A misconfigured robots.txt is the most expensive typo in SEO. One stray slash and Google stops crawling. Worth two minutes of double-checking.
✦ This is 1 of 30+ things overrank automates
A clean robots.txt is the floor. Want overrank to handle the whole technical SEO stack?
Sitemap submission, internal linking, schema markup, image optimization — all on autopilot.
No credit card · takes 60 seconds
Frequently asked questions
What is robots.txt?
robots.txt is a plain-text file that lives at the root of your domain (e.g. https://example.com/robots.txt). It tells search-engine crawlers which paths they are and are not allowed to fetch. Misconfigure it and Google will silently refuse to index your site, no matter how good your content is.
What are the most common robots.txt mistakes?
Three biggies: (1) Disallow: / for User-agent: * — this blocks your entire site from indexing and is the most expensive typo in SEO. (2) Forgetting to point Google at your sitemap with a Sitemap: line. (3) Using underscores in path names where Google expects URL-encoded characters. Our validator catches all three.
Does Google actually obey robots.txt?
Yes, for crawling. Important nuance: robots.txt only stops crawling. It does NOT remove URLs already in Google's index. To remove an indexed URL, use a noindex meta tag or the Search Console URL removal tool — robots.txt alone is not enough.
Should I put a robots.txt on my site if I have nothing to block?
Yes. A minimal robots.txt with just 'User-agent: *' followed by 'Allow: /' and a 'Sitemap:' line tells Google your site is open for indexing and gives it the sitemap URL upfront. A 404 on robots.txt is fine but the explicit allow is friendlier.
Does this tool store my robots.txt?
No. We parse it server-side, return the result, and discard the input immediately. Nothing logged, nothing stored.