SEO & Web · Client-side
Robots.txt Builder
Build and preview your robots.txt visually. Add user-agent rules, block AI crawlers, set crawl delays, and test any URL, all in your browser.
Rules Editor
Sitemap URLs
User-agent: * Allow: /
URL Rule Tester
Test any URL path to see which rule applies and whether it will be crawled.
Tips
- •
/— matches everything from the root - •
/admin/, blocks the entire admin directory - •
/*.pdf$, blocks all PDF files - • Blank Disallow path = allow all (common pattern)
- • Place specific rules before general wildcard (*) blocks
- • Upload to
https://yourdomain.com/robots.txt
Robots.txt Builder — Frequently Asked Questions
01What is a robots.txt file?+
A robots.txt file tells web crawlers which pages or sections of your site they can or cannot access. It lives at your domain root (yourdomain.com/robots.txt) and is checked by Google, Bing, and AI crawlers before they fetch any page. It does not block direct access — it gives instructions that well-behaved bots follow.
02Can I block AI bots like GPTBot and ClaudeBot?+
Yes. The builder includes presets for common AI training crawlers: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (used for Gemini training), CCBot (Common Crawl), and PerplexityBot. Blocking these prevents your content from being used to train AI models, though it does not prevent AI search citations from already-crawled pages.
03What is crawl-delay and when should I use it?+
Crawl-delay tells a bot to wait N seconds between requests. Use it if your server is under stress from aggressive crawling — but note that Googlebot ignores crawl-delay entirely and uses its own crawl rate settings. For Google, adjust crawl rate in Google Search Console instead.
04How does the URL tester work?+
The URL tester runs your current robots.txt rules against any path you enter and shows whether that path would be allowed or blocked for each user-agent you have defined. It checks rules in order — the first matching allow or disallow wins.
05Is this robots.txt builder free?+
Yes, completely free. The builder runs entirely in your browser — no sign-up, no data stored, no server calls. Copy the output and paste it into a robots.txt file at your domain root.