Crawl Budget: When It Actually Matters and How to Fix It

Crawl budget is the amount of crawling Google is willing and able to do on a site. Whether it is worth managing depends on the site, and Google’s own guide draws the line: if a site does not have a large number of pages that change rapidly, or if pages seem to be crawled the same day they are published, “you don’t need to read this guide” (Google’s crawl budget guide). Crawl budget becomes a real problem when a site produces more URLs than Google can or wants to crawl, so that important pages are discovered late or refreshed slowly. The work is first to establish whether that is happening, and only then to reduce waste.

The two components

Google’s guide defines crawl budget by two elements: crawl capacity limit and crawl demand.

Crawl capacity limit (also called hostload) is how much Google can crawl without overloading the server. Google says every site starts with “the same default, conservative crawl capacity limit”, which rises if the site responds quickly and falls if it “slows down” or “responds with server errors (5xx HTTP status codes) or rate-limiting signals (such as HTTP 429)”. The limit is shared across all Google crawlers.

Crawl demand is how much Google wants to crawl. Google lists the factors as perceived inventory, popularity and staleness, and calls perceived inventory “the factor that you can positively control the most”. Site-wide events such as site moves can raise demand temporarily. Google adds that even when the capacity limit is not reached, “if crawl demand is low, Google will crawl your site less”.

A qualification checklist: is crawl budget your problem?

Google’s guide names the sites it is written for, describing the numbers as “a rough estimate” and “not exact thresholds”:

• large sites (1 million or more unique pages) whose content changes moderately often, about once a week;
• medium or larger sites (10,000 or more unique pages) with very rapidly changing content, about daily;
• sites with a large portion of their URLs classified in Search Console as “Discovered, currently not indexed”.

Then check the evidence on your own site:

• [ ] New pages are slow to be crawled. URL Inspection shows important new URLs with no crawl date days or weeks after publication.
• [ ] “Discovered, currently not indexed” is large and growing for URLs you want indexed. Google’s help page says this status typically means Google rescheduled crawling because it expected to overload the site (Page indexing report help).
• [ ] Crawl Stats show crawling concentrated on low-value URLs. Parameter URLs, filter combinations or redirects dominate the sample of crawled URLs.
• [ ] Server response time is high or errors appear in the Crawl Stats host status.
• [ ] The site generates URLs faster than it publishes content, for example through faceted navigation or calendars.

If none of these is true, look elsewhere for the cause of whatever problem you are investigating. On a small site with indexing problems, check quality, duplication and technical causes first; Google’s guide is not written for that case. Google’s Crawl Stats report help says that sites with fewer than a thousand pages “should not need to use this report or worry about this level of crawling detail” (Crawl Stats report help).

Where crawl waste comes from

• Faceted navigation and filters. Google’s faceted navigation guide says URL-parameter facets “can generate infinite URL spaces” (Google on faceted navigation), causing overcrawling and slower discovery of useful URLs.
• Session IDs and tracking parameters that create duplicate URLs.
• Soft 404s. Google’s guide says soft 404 pages “will continue to be crawled, and waste your budget”.
• Long redirect chains and redirected URLs still linked internally.
• Infinite spaces such as calendars with endless next links.
• Duplicate content reachable at many URLs.
• Slow server responses, which lower the capacity limit.

How to fix it

Google’s guide lists the levers:

1. Consolidate duplicates. Remove or canonicalize duplicate URLs so Google spends time on unique content.
2. Block URLs that should not be crawled with robots.txt. Google says blocking “significantly decreases the chance the URLs will be processed by other Google systems”. Note what it adds: Google “won’t shift this newly available crawl budget to other pages unless Google is already hitting your site’s crawl capacity limit”.
3. Do not use noindex to save crawl budget. Google states: “Don’t use noindex, as Google will still request, but then drop the page”.
4. Return 404 or 410 for removed pages. Google calls a 404 “a strong signal not to crawl that URL again”.
5. Eliminate soft 404s.
6. Keep sitemaps current and use accurate lastmod values.
7. Avoid long redirect chains.
8. Make pages faster to load and render. Google notes that if it can load and render pages faster, it might be able to read more content.
9. Monitor crawling with the Crawl Stats report and server logs.

What does not help

• Requesting a higher crawl rate. Google retired the crawl rate limiter tool in Search Console on January 8, 2024 (Google’s announcement), and there is no setting to raise crawling.
• Adding pages to the sitemap repeatedly. A sitemap is a hint, not a crawl order.
• noindex on unwanted URLs (see above).
• Expecting more crawling to improve rankings. Crawling makes indexing possible; it does not make a page rank better.

Measuring before and after

• Crawl Stats report (root-level properties only): total requests, response codes, file types, crawl purpose (discovery vs refresh) and host status. Google notes it counts the URLs actually requested, not canonical URLs, and that example URLs are representative, not complete.
• Server logs: the authoritative record of which URLs verified Googlebot requests. Log analysis is the method for measuring waste precisely.
• Page indexing report: movement in “Discovered, currently not indexed” for URLs you care about.
• URL Inspection: crawl dates for a sample of new and updated important pages.

Filter URLs crowding out 400,000 products

Consider a hypothetical online retailer with 400,000 product pages and color, size and brand filters that produce combination URLs.

• Evidence: Crawl Stats’ sample of crawled URLs is dominated by filter combinations; new products take two to three weeks to show a crawl date; “Discovered, currently not indexed” includes 60,000 product URLs.
• Changes: disallow filter combination patterns in robots.txt while keeping single-filter category pages that have search value crawlable; return 404 for single-filter pages that stay crawlable but list no products (combinations under the disallowed patterns are not fetched, so a 404 there would not be seen); remove redirected URLs from internal links; set lastmod from real product updates.
• Expected result: a larger share of requests reaches product and category pages; new products get crawl dates sooner.
• Verification: compare Crawl Stats and server logs for the four weeks before and after; check crawl dates on newly published products.
• What remains: URLs already known under the blocked patterns may linger in reports; demand, not only capacity, still decides how much Google crawls.

The test for crawl budget work is simple: if Google already crawls important pages promptly, there is nothing to fix. If it does not, remove the URLs that compete with them before trying anything else.