429 Too Many Requests: rate limiting and crawlers

429 Too Many Requests means a client has sent more requests than the server allows in a given time: it has hit a rate limit. It’s how APIs, CDNs and security tools push back against overload, scraping and brute-force attempts, and it’s the one 4xx code Googlebot treats as a reason to slow down rather than as a sign that a page doesn’t exist.

Definition

429 isn’t in the core HTTP spec; it was added by RFC 6585, section 4 in 2012. The response should explain the condition and may include a Retry-After header saying how long to wait, in seconds or as a date. The RFC deliberately doesn’t define how a server identifies the client or counts requests (per IP, per account, per API key), and it says 429 responses must not be stored by caches.

How Googlebot handles 429

Google’s documentation on HTTP status codes says Googlebot treats 429 as a signal that the server is overloaded, the same way it treats a server error. Crawling slows down. Indexed URLs stay in the index for a while, but URLs that keep returning 429 are eventually dropped. If robots.txt itself returns 429, Google temporarily treats the whole site as disallowed.

That makes 429 a legitimate emergency brake. Google’s guide to reducing the crawl rate suggests returning 500, 503 or 429 when Googlebot is overloading your server, but only for a short period, on the order of hours or a day or two. Google retired the Search Console crawl rate limiter in January 2024, so status codes are now the main lever.

Accidental 429s

The more common problem is a rate limit that catches crawlers you want. Typical sources:

• CDN or WAF rate-limiting rules with low thresholds.
• Security plugins that throttle any client making many requests.
• Limits keyed on IP behind a proxy or CDN, where all traffic seems to come from a handful of addresses.
• Hosting plans with request caps.

How to fix it

For site owners

1. Check Search Console’s Crawl Stats report and your CDN or firewall logs for 429s served to Googlebot or Bingbot.
2. Raise thresholds or exempt verified search engine crawlers; verify them by reverse DNS or published IP ranges, not by user-agent.
3. Don’t use 429 as a long-term block. To keep a bot out, use robots.txt or a 403.

For developers

Nginx’s limit_req answers with 503 by default; set it to 429 so clients know it’s a limit, not an outage:

limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/s;

server {
    location / {
        limit_req zone=perip burst=20 nodelay;
        limit_req_status 429;
    }
}

When your own code calls an API that returns 429, back off: honor Retry-After if it’s present, and otherwise retry with growing delays plus some random jitter.

How to check

The HTTP status checker shows whether URLs return 429 to an outside client, and the HTTP header checker reveals the Retry-After value. Compare 503, the code for a server that’s unavailable to everyone.

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 120

{"error":"Rate limit exceeded. Try again in 2 minutes."}

Frequently asked questions