SEO Tools
robots.txt Generator
Choose allow-all, disallow-all or a custom rule set, then add per-user-agent Allow and Disallow directives line by line. Append a sitemap URL and (optionally) a crawl-delay. Copy or download the resulting robots.txt file.
Your input is processed locally in your browser. It is not uploaded to Dueneo.
Loading robots.txt Generator…
How to use
- 1
Pick an access mode
Choose Allow all (open site), Disallow all (block everything) or Custom rules (per-user-agent directives).
- 2
Add custom rules (optional)
In Custom mode, add one block per user-agent. Enter Allow and Disallow paths one per line. Use * to match all bots.
- 3
Add a sitemap URL
The Sitemap: directive at the end of the file tells crawlers where your XML sitemap lives.
- 4
Copy or download
Use Copy to put the robots.txt content on your clipboard, or Download to save it as robots.txt ready to upload to your site root.
Common use cases
- Block crawlers from indexing your /admin/ or /search/ paths.
- Prevent a staging site from being indexed at all (Disallow /).
- Allow Googlebot but slow down Bing and Yandex with a crawl-delay.
- Point crawlers at your sitemap.xml to speed up discovery of new pages.
Limitations
- robots.txt is a request, not a command. Malicious crawlers ignore it.
- Google ignores the
Crawl-delaydirective. Bing and Yandex honour it; this tool emits it only when provided. - Pattern matching (
*,$) is bot-dependent. The tool does not validate patterns — keep them simple to maximise compatibility. - The tool does not verify that your sitemap URL is reachable. Test it in a browser after deploying.