Robots.txt Tester

Robots.txt Tester

Paste your robots.txt file and test whether specific URLs are allowed or blocked for any crawler. Validates syntax against RFC 9309, highlights errors, and extracts sitemap references.

How to Use

Paste your robots.txt content into the input field, enter a URL path to test, and select a User-agent. The tool instantly shows whether the URL is allowed or blocked, which rule matched, and any syntax issues in your robots.txt file.

Features

• RFC 9309 Compliant Parsing – Spec-compliant robots.txt parser with proper directive precedence
• URL Pattern Matching – Full wildcard support with * and $ patterns
• User-Agent Selection – Test against Googlebot, Bingbot, GPTBot, ClaudeBot, and more
• Syntax Validation – Highlights malformed directives, unknown fields, and common mistakes
• Matching Rule Display – Shows exactly which Allow/Disallow rule matched and why
• Sitemap Extraction – Lists all Sitemap directives found in the file
• Crawl-Delay Detection – Identifies crawl-delay settings per User-agent
• Example Robots.txt – Pre-loaded examples for quick testing