Robots.txt Tester
Paste your robots.txt and a list of URLs to test, then pick a user-agent. The tool parses Allow, Disallow, Sitemap and Crawl-delay rules and reports ALLOWED or BLOCKED for each URL using Google's most-specific-match algorithm.
How to use
- 1
Paste your robots.txt
Copy the contents of your robots.txt file into the left textarea. The parser supports User-agent, Allow, Disallow, Crawl-delay and Sitemap directives, plus # comments.
- 2
List URLs to test
Enter one absolute URL per line (e.g. https://example.com/admin/dashboard). The tool extracts the path+query and matches it against the rules.
- 3
Pick a user-agent
Use the dropdown to pick from user-agents declared in the file, or type a custom one. The tool selects the most specific matching group, falling back to *.
- 4
Read the results
Each URL is marked ALLOWED or BLOCKED. The parsed rule groups appear at the bottom so you can verify how the file was interpreted.
Common use cases
- Verify that a staging /admin/ path is blocked before going live.
- Check that wildcard patterns like /*.pdf$ behave as expected.
- Audit a robots.txt inherited from a CMS for unintended blocks.
- Confirm a specific bot (e.g. Googlebot) is allowed on key pages.
Limitations
- The matcher implements Google's most-specific-match rule. Bing and other engines may resolve Allow/Disallow ties differently.
- Wildcard
*and end-anchor$are supported. Other pattern extensions are not. - Crawl-delay is parsed and displayed but has no effect on the allow/block verdict.
- The tool does not fetch live robots.txt files — paste the text.