What is robots.txt?
robots.txt is a plain-text file at the root of a site that tells crawlers which URL paths they may or may not fetch.
robots.txt is a plain-text file served at the root of a host — https://example.com/robots.txt — that tells crawlers which paths they may fetch. After almost three decades as an informal convention, it was standardized as the Robots Exclusion Protocol in RFC 9309 in 2022.
The syntax in one example
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlEach group starts with one or more User-agent lines followed by rules. A crawler obeys only the most specific group that names it, so GPTBot above ignores the * group entirely. Paths are case-sensitive and match from the start of the URL path; * matches any sequence of characters and $ marks the end of the URL.
How Google resolves conflicts
Google applies the longest matching rule. For /search/help, Allow: /search/help beats the shorter Disallow: /search. When an Allow and a Disallow match with equal length, the less restrictive Allow wins. Google ignores crawl-delay and reads only the first 500 KiB of the file — see how Google interprets robots.txt.
Crawling is not indexing
This misunderstanding causes the most damage. robots.txt blocks fetching, not indexing. A disallowed URL that other sites link to can still appear in results, usually without a description. To keep a page out of the index, let it be crawled and use noindex; Google stopped honoring noindex lines inside robots.txt in 2019.
robots.txt and AI search
Most AI companies publish separate user-agents by purpose, so you can opt out of model training while staying visible in AI answers. You might block GPTBot (OpenAI training) but allow OAI-SearchBot (ChatGPT search), or block ClaudeBot while allowing Claude-SearchBot. Google-Extended controls use of your content for Gemini training and grounding; it has no effect on Google Search or AI Overviews. More detail under AI crawler.
Common mistakes
Disallow: /carried over from a staging site into production.- Blocking CSS and JavaScript folders, so Google can’t render the page properly.
- Using robots.txt to hide private areas. The file is public and points straight at what you want hidden — use authentication.
- Assuming one file covers every subdomain. Each host and protocol needs its own.
- A robots.txt that returns a server error. Google treats a 5xx response as “disallow everything” for a while, so crawling stalls.
Test your file
The robots.txt tester fetches and parses your file, tests any URL against Googlebot, Bingbot and AI bots with Google’s longest-match logic, and highlights the exact rule that matched, plus syntax errors and sitemap lines. To understand who is reading the file in the first place, see web crawler.
Frequently asked questions
Does robots.txt remove a page from Google?
No. It only stops crawling. A blocked URL can stay indexed if other pages link to it. Use a noindex meta tag or X-Robots-Tag header, and keep the URL crawlable so search engines can see that instruction.
Do all bots obey robots.txt?
Reputable search engines and most AI companies do, but compliance is voluntary. Some user-triggered fetchers, such as Perplexity-User, may not follow it, and bad bots ignore it, so use authentication or firewall rules for anything that must stay private.
Where must the robots.txt file be placed?
At the root of each host, for example https://www.example.com/robots.txt. A file in a subfolder is ignored, and each subdomain and protocol needs its own file.