Robots.txt Generator & Validator

Robots.txt Generator & Validator

Build spec-compliant robots.txt files with a visual editor, or validate existing ones against RFC 9309. Includes presets for common configurations, syntax highlighting, and a URL tester to check if specific paths are blocked or allowed.

Как использовать

Switch between Generator and Validator tabs. In Generator mode, pick a preset or build custom rules by adding user-agent groups with allow/disallow paths, crawl-delay settings, and sitemap URLs. In Validator mode, paste your robots.txt content to check for syntax errors, then test specific URLs against the rules.

Функции

• Visual Rule Builder – Add multiple user-agent groups with allow/disallow paths, wildcards, and end-of-URL anchors
• Smart Presets – Allow All, Block All, Block AI Bots, WordPress Defaults, E-commerce Template
• RFC 9309 Validation – Syntax checking with line-by-line error and warning highlighting
• URL Tester – Check if a specific URL is blocked or allowed for any user-agent
• Crawl-Delay Support – Set crawl delay per user-agent group
• Multiple Sitemaps – Add multiple sitemap URLs to your robots.txt
• Copy & Download – Copy to clipboard or download as robots.txt file