Robots.txt Validator

What the validator checks

The file is parsed line by line: typos in directives, rules without a leading slash, rules that appear before the first User-agent, a relative sitemap URL, a byte order mark at the start of the file. Directives that not every robot understands are flagged separately, with an explanation of which robots actually read them.

See which rule fired

A "blocked" verdict without an explanation does not help much: you have no idea what to fix. The matching rule and its line number in the file are shown next to the answer. The rule is picked the same way search engines pick it: the longest matching path wins, and when two paths are equally long, Allow beats Disallow.

Differences between robots

The same URL can be blocked for Yandex and allowed for Google. The reason is that a robot with its own block in the file ignores the general block marked with an asterisk. The table shows the answers for all robots side by side instead of one robot at a time.

Test a draft before publishing

You can paste your file contents into the field and check them before putting the file on your site. The parsing is exactly the same; the validator simply does not fetch the file itself.

What to avoid

Blocked styles and scripts prevent Google from seeing the page the way a visitor sees it: Google's own documentation says so. A server error in response to a robots.txt request is worse than having no file at all, because for some time Google treats the whole site as blocked. And the two lines people mix up most: an empty Disallow: allows everything, while Disallow: / blocks the entire site.