robots.txt tester
Search Console's tester checks one URL against the live file. Paste your robots.txt here — including the version you have not deployed yet — and test a whole list of URLs at once, for any user-agent. Each verdict names the rule that decided it, which is the part that makes a surprising block explainable.
What was parsed
How to use
- Paste your
robots.txt. Comments and unknown fields are handled the way crawlers handle them — ignored, but reported in What was parsed so you can see if a directive was misspelled. - Set the user-agent. This matters more than people expect: a
User-agent: Googlebotgroup completely replaces the*group for Googlebot rather than adding to it. - Paste the URLs or paths to test. Full URLs and bare paths both work.
- Read the verdict and the reason. Blocked rows are outlined, and the reason names the rule and why it won.
- Check What was parsed to confirm your groups came out as intended — a stray blank line in the wrong place splits one group into two.
The matching rules, which are not obvious
The most specific user-agent group wins, and only that group applies. If your file has a * group with ten rules and a Googlebot group with one, Googlebot obeys that one rule and ignores all ten. This is the single most common robots.txt mistake: adding a bot-specific group and unintentionally unblocking everything else for it.
The longest matching rule wins, regardless of order. Not first-match, not last-match — longest by character count of the path pattern. So Disallow: /admin/ plus Allow: /admin/public/ means /admin/public/docs is allowed, because the Allow pattern is longer. Rule order in the file is irrelevant.
Allow wins an exact-length tie. If a path matches an Allow and a Disallow of identical length, it is allowed.
Two wildcards are supported — * matches any run of characters, and $ anchors the end of the URL. Disallow: /*.pdf$ blocks PDFs; without the $ it also blocks /file.pdf?download=1. Everything else is a prefix match: Disallow: /admin blocks /administrator too, which is rarely what people mean.
Paths are case-sensitive; the user-agent is not. Disallow: /Admin/ does not block /admin/.
An empty Disallow: allows everything — it is the idiomatic way to say "no restrictions", and is not the same as Disallow: /, which blocks the whole site.
What robots.txt does not do
It controls crawling, not indexing. A URL blocked in robots.txt can still appear in search results — without a snippet, because the crawler was not allowed to read the page — if other sites link to it. To keep a page out of the index you need noindex in a meta tag or the X-Robots-Tag header, and that means the page must be crawlable. Blocking a page in robots.txt and adding noindex to it is self-defeating: the crawler never sees the noindex.
It is also public and advisory. Anyone can read /robots.txt, so listing Disallow: /secret-admin/ advertises the path you were trying to hide, and a crawler that chooses to ignore the file simply does. Use authentication for anything that actually needs protecting.
FAQ
Does this fetch my live robots.txt?
No — paste it in. That is the point: you can test a draft before deploying it, and nothing you paste leaves the tab.
Is the matching the same as Google's?
It implements the algorithm from Google's open-source robots.txt parser and RFC 9309: most-specific user-agent group, longest matching rule, allow wins ties, * and $ wildcards. Bing and most other major crawlers follow the same rules; older or in-house crawlers sometimes use naive first-match, which is stricter.
What about crawl-delay?
Parsed and reported, but it does not affect the allow/block verdict. Google ignores crawl-delay entirely; Bing and Yandex honour it.
Can I test the whole sitemap against it?
Yes, and it is a worthwhile check. Extract the URLs with the sitemap URL lister and paste them here — a URL that is both in your sitemap and blocked in robots.txt is a contradiction that shows up as a warning in Search Console.