robots.txt Generator: Build Your File in a Minute

robots.txt tells crawlers which parts of your site they may fetch. Most sites need only a few lines, but those lines must be exactly right: one misplaced slash can block the whole site, and AI crawlers now add a new decision about training bots versus AI search bots.

This robots.txt generator builds the file from presets — allow all, block admin areas, block or allow AI training bots, allow AI search bots — plus your own custom rules and sitemap lines. Everything happens in your browser, with no signup and no cost.

How to create a robots.txt file

1. Pick a starting preset

   Start from “allow all” or “block admin areas”, then decide how to treat AI training and AI search bots.

2. Add rules and sitemaps

   Add custom Disallow or Allow paths for specific crawlers, plus your sitemap URL.

3. Copy, upload and test

   Save the result as robots.txt in your site’s root folder, then test key URLs before relying on it.

Anatomy of a robots.txt file

# Default rules for all crawlers
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /*?sessionid=

Sitemap: https://example.com/sitemap.xml

• User-agent starts a group and names the crawler; * means any crawler without a group of its own.
• Disallow and Allow take a path starting with /. Paths are case-sensitive prefix matches, with * and $ as wildcards.
• Sitemap takes a full URL and isn’t tied to any group.
• The file is plain UTF-8 text named robots.txt in lowercase, at the root of the host: https://example.com/robots.txt.

The “allow all” preset is the simplest file there is. Both of these mean “crawl everything”, and so do an empty file or no file at all:

User-agent: *
Disallow:

User-agent: *
Allow: /

Note the difference one character makes: Disallow: with nothing after it allows everything, while Disallow: / blocks everything. Google’s guide to creating robots.txt covers the format in full; our robots.txt entry explains the basics.

Choosing an AI bot policy

AI crawlers come in three kinds, and the presets treat them separately:

• AI search bots — OAI-SearchBot, Claude-SearchBot, PerplexityBot — index pages so AI answers can cite and link to them. Blocking them can remove you from those answers.
• User-triggered fetchers — ChatGPT-User, Claude-User, Perplexity-User — read a page when a person asks about it. OpenAI notes that robots.txt may not apply to these requests.
• Training bots and tokens — GPTBot, ClaudeBot, CCBot, plus Google-Extended (Gemini training and grounding) and Applebot-Extended.

A common middle ground allows AI search and blocks training:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /wp-admin/

Search bots aren’t named here, so they follow the * group. Blocking Google-Extended doesn’t affect Google Search or AI Overviews. See how your live file treats each bot with the AI crawler checker, and read about the AI crawler landscape.

Common robots.txt mistakes

• Disallow: / under User-agent: * blocks every compliant crawler from the whole site — a frequent leftover when a staging site goes live.
• Blocking CSS and JavaScript folders that Google needs to render your pages.
• Using robots.txt to hide pages from search. Blocked URLs can still be indexed from links; use noindex on a crawlable page instead.
• Forgetting that named groups ignore *. If you add a Googlebot group, repeat the rules it should still follow.
• Relative sitemap URLs such as Sitemap: /sitemap.xml; use the full address.

After uploading, test real URLs with the robots.txt tester and make sure the sitemap you reference is valid with the sitemap validator.

Tips for a safe robots.txt

• Start minimal; every line you add is a rule you must remember when the site changes.
• Block internal search results and endless filter combinations if they create thousands of near-duplicate URLs — that is where robots.txt saves crawl budget.
• Keep the file under 500 KiB; Google ignores anything beyond that limit.
• Review your AI bot choices once or twice a year, because new crawlers appear and companies split or rename their bots.
• Protect staging sites with a password rather than “Disallow: /”, so the rule can never travel to production.

Limits to keep in mind

The generator writes rules from your choices; it can’t see your site, so it won’t know if a blocked path contains pages you want indexed. robots.txt is voluntary — it guides well-behaved crawlers but blocks nobody by force, so use authentication or firewall rules for that. You upload the file yourself; then test it with the robots.txt tester.

Last updated Oct 30, 2026 by the Yes SEO team.