Robots.txt Tester
Test whether a URL path is allowed or disallowed for a given crawler against pasted robots.txt content. Applies RFC 9309 precedence — longest matching rule wins, Allow beats Disallow on a tie, with `*`/`$` wildcards and most-specific user-agent group selection. Shows the exact rule that decided the verdict.
Input
A path like /admin/dashboard, or a full URL (only its path + query is used).
e.g. Googlebot, Bingbot, GPTBot, ClaudeBot, Google-Extended, or * for the default group.
Output
| Property | Value |
|---|---|
| No data yet | |
Applicable rules (most specific wins)
| Directive | Path pattern | Length | Selected |
|---|---|---|---|
| No data yet | |||
Guides
The Robots.txt Tester checks whether a specific URL path is allowed or blocked for a given crawler, based on the robots.txt rules you paste in. It answers the one question that matters when you are debugging crawling: will this bot be permitted to fetch this URL? — and, just as importantly, it shows you which rule made the decision.
Everything runs in your browser. No request is sent to the site you are testing and nothing is fetched over the network, so you can safely test staging files, draft rules, or a robots.txt you have not published yet.
How to use it
- Paste your
robots.txtcontent into the first box (or load the built-in example). - Enter the URL path you want to test, such as
/admin/dashboard. A full URL works too — only its path and query string are used. - Enter the crawler's user-agent, for example
Googlebot,Bingbot,GPTBot,ClaudeBot, or*for the default group.
The verdict — Allowed or Disallowed — updates instantly, alongside the matched user-agent group, the deciding rule and its line number, any crawl-delay, and a table of every rule that matched the path so you can see exactly why one won.
How the matching works
This tester implements the Robots Exclusion Protocol as standardized in RFC 9309, the same precedence Googlebot uses:
- User-agent groups are case-insensitive. The crawler applies the single group whose product token is the longest match for the tested user-agent, falling back to the
*group when nothing more specific applies. A more specific group (for example, aGooglebotblock) overrides the catch-all*block. - The longest matching rule wins. Specificity is measured by the length of the rule's path pattern — including
*and$— soAllow: /admin/public/beatsDisallow: /admin/for a URL under that public folder. - Allow beats Disallow on a tie. When an
Allowand aDisallowrule of the exact same length both match, access is granted. *and$wildcards are supported:*matches any run of characters and a trailing$anchors the end of the path, soDisallow: /*.pdf$blocks.pdfURLs but not/report.pdfx.- An empty
Disallow:means allow everything — it never blocks a path. - No matching rule means allowed by default.
Why does the tester disagree with my old checker?
A common bug in simpler testers is counting rule length after stripping wildcards, which flips cases like Disallow: /*.pdf$ versus Allow: /page. This tool counts the full pattern length as RFC 9309 specifies, so a wildcard rule keeps the specificity it should have.
Does testing here reflect what Google actually crawls?
The allow/disallow logic mirrors Google's public documentation and RFC 9309. Keep in mind that robots.txt only governs crawling, not indexing — a disallowed URL can still appear in search results if it is linked elsewhere. Use noindex to keep a page out of the index.
Use it from code
From 3 credits per callREST API
curl -X POST https://api.iotools.cloud/v1/tool/robots-txt-tester \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"robots": "User-agent: *\nDisallow: /admin/\nAllow: /admin/pu…",
"testUrl": "/admin/dashboard",
"userAgent": "Bingbot"
}'Swap in your own key from your account. The tool's fields are the body — no wrapper.
Ask an AI agent
Use the IOTools `robots-txt-tester` tool (Robots.txt Tester) on this input:
YOUR_INPUT_HEREPaste this at any agent connected to the IOTools MCP server, then add your input.