# robots.txt Generator: Build Your File in a Minute

> Free robots.txt generator: build a valid file with presets to block admin areas, block or allow AI training bots and allow AI search bots, plus sitemap lines.

Pick a preset, add your own rules and sitemap, and copy a valid robots.txt ready to upload.

- Use it free (no signup): https://yes-seo.com/robots-txt-generator
- What it does: robots.txt generator: build a robots.txt with presets (allow all, block admin areas, block or allow AI training bots, allow AI search bots), custom rules and sitemap lines.

robots.txt tells crawlers which parts of your site they may fetch. Most sites need only a few lines, but those lines must be exactly right: one misplaced slash can block the whole site, and AI crawlers now add a new decision about training bots versus AI search bots.

This robots.txt generator builds the file from presets — allow all, block admin areas, block or allow AI training bots, allow AI search bots — plus your own custom rules and sitemap lines. Everything happens in your browser, with no signup and no cost.

## How to create a robots.txt file

1. **Pick a starting preset** — Start from “allow all” or “block admin areas”, then decide how to treat AI training and AI search bots.
2. **Add rules and sitemaps** — Add custom Disallow or Allow paths for specific crawlers, plus your sitemap URL.
3. **Copy, upload and test** — Save the result as robots.txt in your site’s root folder, then test key URLs before relying on it.

## Anatomy of a robots.txt file

```
# Default rules for all crawlers
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /*?sessionid=

Sitemap: https://example.com/sitemap.xml
```

- `User-agent` starts a group and names the crawler; `*` means any crawler without a group of its own.

- `Disallow` and `Allow` take a path starting with `/`. Paths are case-sensitive prefix matches, with `*` and `$` as wildcards.

- `Sitemap` takes a full URL and isn’t tied to any group.

- The file is plain UTF-8 text named `robots.txt` in lowercase, at the root of the host: `https://example.com/robots.txt`.

The “allow all” preset is the simplest file there is. Both of these mean “crawl everything”, and so do an empty file or no file at all:

```
User-agent: *
Disallow:

User-agent: *
Allow: /
```

Note the difference one character makes: `Disallow:` with nothing after it allows everything, while `Disallow: /` blocks everything. Google’s [guide to creating robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt) covers the format in full; our [robots.txt](https://yes-seo.com/glossary/robots-txt) entry explains the basics.

## Choosing an AI bot policy

AI crawlers come in three kinds, and the presets treat them separately:

- **AI search bots** — `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot` — index pages so AI answers can cite and link to them. Blocking them can remove you from those answers.

- **User-triggered fetchers** — `ChatGPT-User`, `Claude-User`, `Perplexity-User` — read a page when a person asks about it. OpenAI notes that robots.txt may not apply to these requests.

- **Training bots and tokens** — `GPTBot`, `ClaudeBot`, `CCBot`, plus `Google-Extended` (Gemini training and grounding) and `Applebot-Extended`.

A common middle ground allows AI search and blocks training:

```
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /wp-admin/
```

Search bots aren’t named here, so they follow the `*` group. Blocking `Google-Extended` doesn’t affect Google Search or AI Overviews. See how your live file treats each bot with the [AI crawler checker](https://yes-seo.com/ai-crawler-checker), and read about the [AI crawler](https://yes-seo.com/glossary/ai-crawler) landscape.

## Common robots.txt mistakes

- **`Disallow: /` under `User-agent: *`** blocks every compliant crawler from the whole site — a frequent leftover when a staging site goes live.

- **Blocking CSS and JavaScript** folders that Google needs to render your pages.

- **Using robots.txt to hide pages from search.** Blocked URLs can still be indexed from links; use `noindex` on a crawlable page instead.

- **Forgetting that named groups ignore `*`.** If you add a `Googlebot` group, repeat the rules it should still follow.

- **Relative sitemap URLs** such as `Sitemap: /sitemap.xml`; use the full address.

After uploading, test real URLs with the [robots.txt tester](https://yes-seo.com/robots-txt-tester) and make sure the sitemap you reference is valid with the [sitemap validator](https://yes-seo.com/sitemap-validator).

## Tips for a safe robots.txt

- Start minimal; every line you add is a rule you must remember when the site changes.
- Block internal search results and endless filter combinations if they create thousands of near-duplicate URLs — that is where robots.txt saves crawl budget.
- Keep the file under 500 KiB; Google ignores anything beyond that limit.
- Review your AI bot choices once or twice a year, because new crawlers appear and companies split or rename their bots.
- Protect staging sites with a password rather than “Disallow: /”, so the rule can never travel to production.

## Limits to keep in mind

The generator writes rules from your choices; it can’t see your site, so it won’t know if a blocked path contains pages you want indexed. robots.txt is voluntary — it guides well-behaved crawlers but blocks nobody by force, so use authentication or firewall rules for that. You upload the file yourself; then test it with the [robots.txt tester](https://yes-seo.com/robots-txt-tester).


## Frequently asked questions

### Where do I upload robots.txt?

To the root of your host, so it loads at https://yourdomain.com/robots.txt. Each subdomain needs its own file, and a robots.txt in a subfolder is ignored.

### What is the best robots.txt for SEO?

For most small sites, a short file that allows everything except admin or cart areas and lists the sitemap. Block more only when you have a clear reason, such as endless filter URLs.

### How do I block AI bots with robots.txt?

Add a group naming the bots, such as GPTBot, ClaudeBot and CCBot, with “Disallow: /”. Decide separately about AI search bots like OAI-SearchBot, because blocking them can remove you from AI search answers.

### Can robots.txt stop Google from indexing a page?

It stops crawling, not indexing. To remove a page from results, let Google crawl it and add a noindex meta tag or X-Robots-Tag header.

### Do I need a robots.txt file at all?

No. Without one, crawlers assume they may fetch everything, and a missing file (404) is fine. Add one when you need to block areas or want to list your sitemap.