5xx Server Errors: How They Affect Crawling and How to Fix Them

A 5xx response tells a crawler that the server failed to handle a valid request. Google’s response is predictable and documented: its crawlers slow down, any content returned with the error is ignored, and URLs that keep returning server errors are eventually dropped from the index. Google’s HTTP status documentation states that 5xx and 429 errors “prompt Google’s crawlers to temporarily slow down with crawling”, that already indexed URLs “are preserved in the index, but eventually dropped”, and that once the server returns 2xx again, Google gradually increases the crawl rate (Google’s HTTP status code documentation). Short, rare errors are survivable. Persistent or repeated ones cost crawling and, in the end, indexed pages.

Codes are not interchangeable

Different failures look similar in a dashboard but have different causes and different consequences. Keep them separate.

Two distinctions matter most:

• 403 is not 5xx. A firewall refusing Googlebot produces 4xx responses; Google’s Page indexing help notes that Googlebot never provides credentials, so a 403 to Googlebot is returned incorrectly (Page indexing report help). The fix is in the security configuration, not server capacity.
• 429 slows crawling like a 5xx. Rate limiting that catches Googlebot reduces crawl capacity even though the server is healthy.

The robots.txt special case

When robots.txt itself returns a server error, the impact is site-wide. Google’s robots.txt specification says that for the first 12 hours Google stops crawling the site while it keeps trying to fetch the file, then uses the last good version for up to 30 days. Google’s guidance on pausing a site warns against returning 503 for robots.txt “because this blocks all crawling” (Google on temporarily pausing a site). A misconfigured maintenance mode that serves 503 on every path, including robots.txt, stops crawling of the whole site.

Temporary errors vs repeated failure

A short outage. A few minutes or hours of 5xx during an incident slows crawling temporarily. Google preserves indexed URLs during that time. When the server recovers, crawling recovers gradually.

Planned downtime. Google’s guidance: if you need to disable a site urgently for one to two days, return an informational error page with a 503 status. It cautions that a 503 should be used only for a very short period, “a few days at most”, because URLs will otherwise be dropped from the index. Keep robots.txt returning 200 throughout.

Repeated failure. Errors that recur every night, every traffic peak or on specific templates keep sending the signals Google’s crawl budget guide says lower the crawl capacity limit. The limit falls, discovery of new pages slows, and URLs that keep failing are removed.

Diagnosis workflow

1. Confirm the scope in Search Console.

• Crawl Stats: the host status section and the breakdown by response show server errors over time.
• Page indexing report: “Server error (5xx)” lists affected URLs with examples.

2. Match with server logs. Find the verified Googlebot requests that received 5xx and note timestamps, paths and the server that answered (CDN edge or origin).

1. Separate classes of failure. Group by status code, by layer (CDN, load balancer, origin, application) and by URL pattern.
2. Look for timing patterns. Errors at fixed times point to scheduled jobs, backups or cache rebuilds. Errors during traffic peaks point to capacity.
3. Check the security layer. A WAF or bot manager can return 429, 403 or custom 5xx pages to crawlers it considers suspicious. Its event logs show which rule fired.
4. Reproduce. Request the failing URLs directly at the reported times, with and without a CDN, and check response headers.

Fixes by cause

Verifying the fix

Do not declare an access problem fixed because one request succeeds. Verify across time:

• server logs show verified Googlebot requests to previously failing URLs returning 200 across several days, including the hours when failures used to occur;
• Crawl Stats host status returns to normal and server error counts fall;
• the Page indexing report’s “Server error (5xx)” count declines after recrawl, and Validate fix can be used once fixes are in place.

Tracing nightly 502 errors on a news site

Suppose that on a news site, Search Console shows a spike in “Server error (5xx)” and Crawl Stats shows total requests falling by a third.

• Logs: 502 responses to verified Googlebot between 01:00 and 03:00 each night; human traffic in those hours sees the same errors.
• Cause: a nightly sitemap and cache rebuild saturates the origin’s database; the CDN returns 502 when the origin does not respond.
• Fix: move the rebuild to an incremental process, add a read replica, and serve stale cache on origin errors.
• Verification: check that two weeks of logs show 200 responses during the former error window, that Crawl Stats requests return toward the previous level, and that the 5xx list shrinks after recrawl.
• What remains: some URLs that were dropped during the failures need recrawling before they reappear.

What the access evidence must show before calling it a 5xx problem

Automated tools and third-party monitors can fail to fetch a page for reasons that have nothing to do with the server’s response to Googlebot: their own network, a bot challenge, DNS from their location. Before classifying a problem as server errors to Google, collect the request time, client, status code if one was returned, response headers, and the matching server or CDN log entry. A failed fetch without a status code is a network or client failure until proven otherwise.