Robots.txt tester and validator
One wildcard in robots.txt can block a whole section by accident. Test the URLs that matter before you publish a change.
Online checks send only the address to the IndexFix Tools server, which downloads that public page and passes the answer back. Nothing is stored.
Your data stays in your browser. Nothing is uploaded or stored.
Key facts
- Google supports the user-agent, allow, disallow and sitemap fields; other fields such as crawl-delay aren't supported.
- Google uses the most specific rule based on the length of the rule path; with conflicting rules it uses the least restrictive one.
- Rule paths are case-sensitive, and Google ignores content after the first 500 KiB of the file.
- robots.txt isn't a way to keep a page out of Google: a blocked URL can still be indexed if other pages link to it.
Download the checklist: diagnosing indexing problems
Step by step: status in Search Console, page availability, robots.txt, noindex, canonical URL, redirects, sitemap and backlinks, with what to do in each case.
Download PDF PDF · 2 pagesHow to use it
- Open your-site/robots.txt in the browser and copy the whole file (or paste a draft you plan to publish).
- Paste it into the first box.
- Enter the user-agent token to test, for example Googlebot.
- Paste the URLs or paths to test, one per line.
- Click Test URLs and read the result and the deciding line for each URL.
Where to get the data
The live robots.txt
Always at the root of the host: https://your-site/robots.txt. Each host and protocol has its own file.
Search Console
The robots.txt report in Search Console settings shows whether Google can process your robots.txt files. Use it to confirm the live state after a change.
URLs to test
Important templates: the home page, category and product pages, parameter URLs, CSS and JavaScript files, and anything you plan to block.
How Google picks a group
A crawler follows only one group: the one whose user agent matches its own most specifically. Groups with the same user agent are combined. If no group names the crawler, the * group applies; if there is none, everything is allowed.
How rules are matched
Rules are compared with the path of the URL, including the query string. * matches any sequence of characters and $ marks the end of the URL. The longest matching rule wins, and if an allow and a disallow rule are equally long, the allow rule wins.
What a block does and doesn't do
A disallowed URL isn't crawled, so Google can't read its content or its meta tags, including noindex. It can still appear in results without a description if other pages link to it. To keep a page out of search, allow crawling and use noindex.
Next step
Blocked pages are only half the story: find stray noindex tags in bulk as well.
Questions and answers
Why doesn't the tester fetch my robots.txt itself?
Browsers don't let one site read files from another site without permission, so the file has to be pasted. That also lets you test a draft before publishing it.
Does it follow Googlebot-Image or other crawlers?
Enter any user-agent token. The tester uses the group that names that token, or the * group if none does.
Are lines like Crawl-delay checked?
They're counted as unsupported and ignored, as Google ignores them.
Does Disallow remove a page from Google?
No. It stops crawling. To remove a page from search, allow crawling and add noindex, or use the Removals tool for urgent cases.
Sources
- How Google interprets the robots.txt specification — Google Search Central: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt
- Introduction to robots.txt — Google Search Central: https://developers.google.com/search/docs/crawling-indexing/robots/intro