AI Crawler Checker: Which AI Bots Can Reach Your Site?
Enter a URL to see, bot by bot, whether search, user-fetch and training crawlers are allowed — and whether your firewall really lets them through.
Each AI company now runs several bots with different jobs. OpenAI’s OAI-SearchBot finds pages for ChatGPT search, ChatGPT-User fetches a page when someone asks about it, and GPTBot collects training data. Anthropic, Perplexity, Apple and Google split their crawlers in a similar way, so a single “allow AI” or “block AI” rule is a blunt instrument.
This AI crawler checker reads your robots.txt and tests it against more than 25 AI user-agents, grouped by purpose. It then requests your page with real AI user-agent strings to catch firewall or CDN rules that answer 403 even when robots.txt says yes.
How to check AI crawler access
Enter your page URL
We fetch robots.txt from the root of that host and test the exact URL you entered.
Review the access table
Each bot shows allowed or blocked, the rule that decided it and whether it is a search, user-fetch or training bot.
Check the live test
A bot allowed by robots.txt that still gets a 403 or a challenge page is blocked by your firewall, CDN or host.
AI bots by purpose
| Company | Search / index | User-triggered fetch | Training |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT-User | GPTBot |
| Anthropic | Claude-SearchBot | Claude-User | ClaudeBot |
| Perplexity | PerplexityBot | Perplexity-User | — |
| Googlebot (Search, AI Overviews) | — | Google-Extended (Gemini training and grounding) | |
| Apple | Applebot | — | Applebot-Extended |
The checker also covers CCBot (Common Crawl), Meta-ExternalAgent, Bytespider, Amazonbot, DuckAssistBot, MistralAI-User and others.
Two tokens behave differently. Google-Extended and Applebot-Extended never visit your site; they are robots.txt tokens that control how content fetched by Googlebot or Applebot may be used. Blocking Google-Extended does not affect Google Search rankings or AI Overviews, as Google’s crawler documentation states.
User-triggered fetchers act for a person who asked a question. OpenAI notes that robots.txt rules may not apply to ChatGPT-User, and Perplexity-User may not follow them either. More background: AI crawler.
Block training, stay in AI search
A common policy is to keep AI search and user fetches open while opting out of training:
# AI search and user fetches: allowed, admin still private
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Disallow: /admin/
# AI training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
# Everyone else
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlNotice that Disallow: /admin/ appears twice. A crawler obeys only the most specific group that names it, so a bot with its own group ignores everything under User-agent: *. Build a file like this with the robots.txt generator and test single URLs with the robots.txt tester.
When robots.txt says yes but the bot is still blocked
robots.txt is a request, not a gate. The gate is your server, and that is where most accidental AI blocks happen:
- CDN bot protection. CDNs and hosts offer settings that block AI crawlers or challenge unknown bots, answering with 403 Forbidden or a challenge page.
- WAF rules and security plugins that block unfamiliar user-agents or data-center IP ranges.
- Rate limits that return 429 Too Many Requests when a bot fetches several pages quickly.
The live test reports the status code each AI user-agent receives. If GPTBot is allowed in robots.txt but gets a 403, look at your firewall, not your robots.txt. Your access logs show what the real bots received:
# Status codes served to AI bots (Apache/Nginx combined log format)
grep -E 'GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot' access.log \
| awk '{print $9}' | sort | uniq -c
412 200
37 403 <- something is blocking these requestsAnyone can fake a user-agent, so compare the IP addresses in your logs with the ranges the companies publish, such as those listed by Perplexity and OpenAI, before trusting a line that claims to be a bot.
Tips for managing AI crawlers
- Decide per purpose, not per company: search and user-fetch bots can send visitors, training bots don’t.
- Repeat important Disallow rules inside every named group, because a bot with its own group ignores User-agent: *.
- Re-check after changing CDN or security-plugin settings; they often block AI bots without touching robots.txt.
- Allow a day for changes: OpenAI and Perplexity both say robots.txt updates can take about 24 hours to take effect.
- Use a firewall rule, not robots.txt, when you need a hard block — robots.txt only works for bots that choose to obey it.
Limits to keep in mind
robots.txt results show what each bot is asked to do; whether it complies is up to its operator. The live test sends genuine AI user-agent strings from our servers, not from the companies’ own IP ranges, so a firewall that verifies bots by IP may treat our test differently from the real crawler — confirm in your server logs. Each run tests one URL; for wider coverage, test key sections with the robots.txt tester or run the AI SEO checker.
Last updated Oct 30, 2026 by the Yes SEO team.
AI Crawler Checker: frequently asked questions
What is an AI crawler?
A bot that fetches web pages for an AI product: to build a search index for AI answers, to read a page a user asked about, or to collect training data. Each has its own user-agent token you can address in robots.txt.
Should I block GPTBot?
That is a business decision. GPTBot only collects training data, and blocking it doesn’t stop OAI-SearchBot from showing your pages in ChatGPT search, because OpenAI treats them as independent settings.
Does blocking Google-Extended hurt my Google rankings?
No. Google states that Google-Extended doesn’t affect inclusion or ranking in Google Search. It only controls use of your content for Gemini training and grounding.
Why is a bot blocked when my robots.txt allows it?
Usually a CDN, firewall or security plugin returns 403 to that user-agent or IP range. The live test shows the status code each bot receives, so you can tell the two causes apart.
Do AI bots obey robots.txt?
The major search and training crawlers say they do. User-triggered fetchers are different: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity-User may not follow it.
How many AI bots does the checker test?
More than 25, including OpenAI, Anthropic, Perplexity, Google, Apple, Meta, ByteDance, Amazon, DuckDuckGo, Mistral and Common Crawl agents. Each result is labeled as a search, user-fetch or training bot, so you see at a glance what you allow.