# What is robots.txt?

> robots.txt is a plain-text file at the root of a site that tells crawlers which URL paths they may or may not fetch.

URL: https://yes-seo.com/glossary/robots-txt

**robots.txt** is a plain-text file served at the root of a host — `https://example.com/robots.txt` — that tells crawlers which paths they may fetch. After almost three decades as an informal convention, it was standardized as the Robots Exclusion Protocol in [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) in 2022.

## The syntax in one example

```
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml
```

Each group starts with one or more `User-agent` lines followed by rules. A crawler obeys only the most specific group that names it, so GPTBot above ignores the `*` group entirely. Paths are case-sensitive and match from the start of the URL path; `*` matches any sequence of characters and `$` marks the end of the URL.

## How Google resolves conflicts

Google applies the **longest matching rule**. For `/search/help`, `Allow: /search/help` beats the shorter `Disallow: /search`. When an Allow and a Disallow match with equal length, the less restrictive Allow wins. Google ignores `crawl-delay` and reads only the first 500 KiB of the file — see [how Google interprets robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt).

## Crawling is not indexing

This misunderstanding causes the most damage. robots.txt blocks *fetching*, not *indexing*. A disallowed URL that other sites link to can still appear in results, usually without a description. To keep a page out of the index, let it be crawled and use [noindex](https://yes-seo.com/glossary/noindex); Google stopped honoring noindex lines inside robots.txt in 2019.

## robots.txt and AI search

Most AI companies publish separate user-agents by purpose, so you can opt out of model training while staying visible in AI answers. You might block `GPTBot` (OpenAI training) but allow `OAI-SearchBot` (ChatGPT search), or block `ClaudeBot` while allowing `Claude-SearchBot`. `Google-Extended` controls use of your content for Gemini training and grounding; it has no effect on Google Search or AI Overviews. More detail under [AI crawler](https://yes-seo.com/glossary/ai-crawler).

## Common mistakes

- `Disallow: /` carried over from a staging site into production.

- Blocking CSS and JavaScript folders, so Google can’t render the page properly.

- Using robots.txt to hide private areas. The file is public and points straight at what you want hidden — use authentication.

- Assuming one file covers every subdomain. Each host and protocol needs its own.

- A robots.txt that returns a server error. Google treats a 5xx response as “disallow everything” for a while, so crawling stalls.

## Test your file

The [robots.txt tester](https://yes-seo.com/robots-txt-tester) fetches and parses your file, tests any URL against Googlebot, Bingbot and AI bots with Google’s longest-match logic, and highlights the exact rule that matched, plus syntax errors and sitemap lines. To understand who is reading the file in the first place, see [web crawler](https://yes-seo.com/glossary/crawler).

## Frequently asked questions

### Does robots.txt remove a page from Google?

No. It only stops crawling. A blocked URL can stay indexed if other pages link to it. Use a noindex meta tag or X-Robots-Tag header, and keep the URL crawlable so search engines can see that instruction.

### Do all bots obey robots.txt?

Reputable search engines and most AI companies do, but compliance is voluntary. Some user-triggered fetchers, such as Perplexity-User, may not follow it, and bad bots ignore it, so use authentication or firewall rules for anything that must stay private.

### Where must the robots.txt file be placed?

At the root of each host, for example https://www.example.com/robots.txt. A file in a subfolder is ignored, and each subdomain and protocol needs its own file.