What is Web crawler?

A web crawler — also called a spider or bot — is a program that fetches web pages automatically, extracts their links and adds the new URLs to a queue to fetch next. Search engines run crawlers to build their indexes; AI companies run them to feed answer engines and to train models.

How a crawler works

1. Seed URLs come from pages it already knows, from sitemaps, and from links found elsewhere.
2. Politeness checks: before fetching, the crawler reads the host’s robots.txt and skips disallowed paths.
3. Fetching: it requests the URL and records the status code, headers and HTML.
4. Parsing: it extracts links from <a href> elements, plus canonicals and other signals.
5. Scheduling: new and changed URLs are prioritized, and servers that slow down receive fewer requests.

Google adds a rendering step for JavaScript later in its pipeline. Many other crawlers don’t, which is why content in the server HTML matters so much.

The crawlers that matter

• Googlebot — mostly its smartphone crawler, because Google indexes the mobile version of pages.
• Bingbot — Bing’s index also feeds several other search products.
• AI crawlers, split by purpose. Search bots such as OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages for answers that can cite you. Training bots such as GPTBot, ClaudeBot and CCBot collect data for models. User-triggered fetchers such as ChatGPT-User and Claude-User load a page because someone asked about it. More under AI crawler.

Example: Googlebot in your access log

66.249.66.1 - - [28/Oct/2026:09:14:02 +0000] "GET /pricing HTTP/1.1" 200 18432 "-"
"Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Anyone can fake that user-agent string. To confirm a real Googlebot, run a reverse DNS lookup on the IP address — it should resolve to googlebot.com, google.com or googleusercontent.com — then a forward lookup back to the same IP, or compare it with Google’s published IP ranges (verifying Googlebot). Google shows the Chrome version as W.X.Y.Z because it changes as the crawler’s browser updates.

Common mistakes

• Firewall rules that block good bots. Aggressive bot protection at a CDN can return 403 to Googlebot or AI search crawlers while robots.txt says they are welcome.
• Trusting the user-agent alone. Bad actors simply lie about who they are; verify by IP for anything important.
• Links crawlers can’t follow: buttons with onclick handlers, anchors without href, URLs assembled only in JavaScript. See JavaScript SEO.
• Endless URL spaces from calendars and filters that trap crawlers and waste crawl budget.

Test crawler access

The robots.txt tester checks any URL against the rules for Googlebot, Bingbot and AI bots. To catch firewall blocks as well, the AI crawler checker requests your page with real AI user-agents and reports any 403 responses.

Frequently asked questions