About this robots.txt generator
How to use it
- Pick a preset: Allow all, Disallow all, Block all AI bots, WordPress, or Shopify — or start from the default and edit freely.
- Toggle the AI crawlers you want to block (GPTBot, ClaudeBot, CCBot, and 7 more) — each checked bot gets its own User-agent group.
- Add custom rules: choose a user-agent, Allow or Disallow, and the path. Use * as a wildcard and a trailing $ to anchor the end.
- Add your sitemap URL and, if needed, a crawl-delay. Watch the live preview and fix any syntax warnings the checker flags.
- Test a path — "will Googlebot fetch /admin/?" — then Copy or Download robots.txt and upload it to your domain root.
The method behind it
Frequently asked questions
How do I block GPTBot and ClaudeBot in robots.txt?
Give each bot its own group with Disallow: / — write "User-agent: GPTBot" on one line and "Disallow: /" on the next, then repeat for ClaudeBot. This tool's "Block all AI bots" preset does it for ten crawlers at once (GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, PerplexityBot, Bytespider, Meta-ExternalAgent). One honest caveat: robots.txt is a voluntary standard. Reputable crawlers honor it, but a bad actor can ignore it — treat it as an opt-out signal, not access control.
What is the difference between "allow all" and "disallow all"?
"Allow all" is User-agent: * followed by an empty Disallow: — an empty value matches nothing, so every crawler may fetch everything. "Disallow all" is User-agent: * with Disallow: / — the single slash matches every path on the site. Use "disallow all" on staging sites you never want indexed, and switch to your real rules at launch.
Where do I put the robots.txt file?
At the root of your domain, so it is served at https://example.com/robots.txt. Crawlers only check that one location — a robots.txt inside a subfolder like /blog/robots.txt is ignored completely. After uploading, open the URL in your browser and confirm it serves as plain text with a 200 status.
Does Google honor the crawl-delay directive?
No. Googlebot ignores crawl-delay entirely — if Google crawls too aggressively, adjust the crawl rate in Google Search Console instead. A few smaller crawlers do honor crawl-delay, which is why the tool still offers it, but treat it as a polite request, never a limit.
How can I test my robots.txt before uploading?
Use the built-in tester above: pick a crawler, enter a path, and it applies Google's matching rules — longest user-agent match wins, longest path rule wins, ties go to Allow — and returns an ALLOWED or BLOCKED verdict with the exact rule that matched. After uploading, double-check with Google Search Console's robots.txt report, which shows precisely how Googlebot reads your live file.
Worked example
Goal: a new WordPress blog wants Google traffic but no AI training on its posts.
Setup: choose the WordPress preset, leave all ten AI crawlers checked, and enter the sitemap https://example.com/sitemap.xml.
Output (excerpt):
User-agent: *→Disallow: /wp-admin/,Allow: /wp-admin/admin-ajax.php,Disallow: /wp-includes/…User-agent: GPTBot→Disallow: /(one group like this per AI crawler)Sitemap: https://example.com/sitemap.xml
Tester checks:
- GPTBot →
/hello-world/→ BLOCKED (matchedDisallow: /) - Googlebot →
/hello-world/→ ALLOWED (no rule matched) - Googlebot →
/wp-admin/→ BLOCKED (matchedDisallow: /wp-admin/)
Upload the finished file to https://example.com/robots.txt — a robots.txt only takes effect at the domain root.