Robots.txt Generator
Generate a valid robots.txt file: a default rule group for all crawlers, an optional rule group for one specific bot (Googlebot, GPTBot, etc.), Crawl-delay, and one or more Sitemap URLs. Produces spec-correct output ready to copy or download.
Input
Written as one or more “# …” comment lines at the top of the file.
Default Rule Group — User-agent: *
One path per line. Must start with “/”. Leave both boxes empty to allow everything.
One path per line. Use to carve out an exception inside a disallowed path.
Additional Rule Group (optional)
Adds a second, more specific rule group above the default one.
Only used when a User-agent is selected above.
Sitemap
One absolute URL per line (must start with http:// or https://).
Output
Guides
Build a valid robots.txt file without memorizing the syntax. Set the rules for all crawlers, optionally add a stricter (or looser) rule group for one specific bot, list your sitemap URLs, and copy or download the result — ready to drop into the root of your site.
What is robots.txt?
robots.txt is a plain-text file at the root of a domain (https://example.com/robots.txt) that tells well-behaved crawlers which parts of a site they may or may not request — the first thing most search engine and AI bots check before crawling. It is not an access-control mechanism, just an advisory convention (RFC 9309): it won't stop a determined scraper, and it doesn't hide a URL from search results if another page links to it (use a noindex meta tag for that).
How to use this generator
- Default rule group — list the paths every crawler should skip under Disallow paths (one per line, starting with
/, e.g./admin/), and any exceptions under Allow paths. Leave both empty to allow crawling everywhere. - Additional rule group (optional) — pick a specific crawler (Googlebot, GPTBot, Bingbot, or a custom user-agent) to give it its own rules, separate from the default group. Common use: allow your main search bot everywhere while blocking AI training crawlers entirely.
- Set a Crawl-delay in seconds if a bot is hitting your server too aggressively (most major engines besides Bing/Yandex ignore this, but it's harmless to include).
- Add one or more Sitemap URLs — full absolute URLs (
https://example.com/sitemap.xml), not relative paths — so crawlers can find your sitemap directly from robots.txt. - Copy the output or download it as
robots.txt, then upload it to your site's document root.
How rules are written
Each rule group starts with a User-agent: line, followed by its Disallow:/Allow:/Crawl-delay: directives. If a group has no Disallow or Allow paths at all, the generator writes an explicit empty Disallow: line — per the spec this means "allow everything," which is clearer than a group with no directives at all.
Where does robots.txt need to go?
At the root of your domain only — https://example.com/robots.txt. A copy at https://example.com/blog/robots.txt is ignored; crawlers only ever check the root.
Does robots.txt block a page from search results?
Not reliably — a linked-to URL can still be indexed with no snippet. To keep a page out of results entirely, allow it to be crawled and use a noindex meta tag or header instead.
Can I block AI crawlers specifically?
Yes — add an additional rule group for the bot's user-agent (e.g. GPTBot, Google-Extended, CCBot, ClaudeBot) with Disallow: /, while leaving your default * group open for search engines.
How do I check whether a URL is actually blocked?
Use the companion Robots.txt Tester — paste robots.txt content plus a URL and user-agent to see the exact verdict and the rule that decided it, using the same longest-match precedence real crawlers apply.
Is this private?
Yes. Generation happens entirely in your browser — nothing you type is uploaded to a server.