Log File Analysis for SEO: What Googlebot Is Actually Doing on Your Site
- Filed November 10, 2026
- Crawling and indexing, guide 8 of 16
Server access logs record every request that reaches the server: which URLs search engine crawlers requested, when, and what the server answered. Search Console’s Crawl Stats report summarizes Google’s crawling, but Google describes its example URLs as “not comprehensive, but just a representative example” (Crawl Stats report help). Logs show every request that reached the layer writing them. Log analysis turns them into answers to practical questions: are important pages being crawled, how much crawling goes to URLs that should not exist, which errors does Googlebot receive, and how quickly are new pages discovered.
What a log line contains
Web servers write one line per request. A line in the combined log format, a standard format in Apache and Nginx configurations, looks like this (illustrative record):
66.249.66.10 - - [04/Nov/2026:09:14:22 +0000] "GET /products/trail-shoe-42/ HTTP/1.1" 200 48211 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"| Field | Value in the example | Why it matters |
|---|---|---|
| Client IP | <!–INLINECODE0–> | Needed to verify the crawler |
| Timestamp | <!–INLINECODE1–> | Crawl timing and frequency |
| Request | <!–INLINECODE2–> | Which URL was requested |
| Status | <!–INLINECODE3–> | What the crawler received |
| Bytes | <!–INLINECODE4–> | Response size |
| Referrer | <!–INLINECODE5–> | Empty in this record |
| User agent | <!–INLINECODE6–> | Claimed identity |
CDN and load-balancer logs may use other formats and may record the edge’s response rather than the origin’s. Decide which layer’s logs represent what crawlers actually received.
Step 1: verify the crawler
A user-agent string alone does not authenticate a crawler; any client can send Googlebot in a header. Google documents a verification method (verifying Google’s crawlers):
- Run a reverse DNS lookup on the IP address from the log.
- Confirm the domain name is
googlebot.com,google.comorgoogleusercontent.com. - Run a forward DNS lookup on that domain name.
- Confirm it returns the original IP address.
Google also publishes IP ranges as JSON files for its common crawlers, special-case crawlers and user-triggered fetchers, which allows verification at scale without DNS lookups for every line. Other providers publish their own: OpenAI publishes IP lists for its agents (OpenAI’s crawler documentation), and Anthropic says a source IP on its published list indicates the request came from Anthropic (Anthropic’s crawler documentation).
Separate verified crawler traffic from impostors before any analysis. Impostor requests do not describe Google’s behavior.
Step 2: segment the requests
Group verified Googlebot requests by:
- Site section or template (products, categories, articles, parameters, assets).
- Status code (200, 301/302, 304, 404/410, 5xx, 429).
- Crawler type (smartphone vs desktop; Googlebot vs Googlebot-Image and others).
- URL type (canonical, parameter variant, redirecting URL, non-canonical duplicate).
- Time (day, hour) to see patterns and spikes.
Step 3: answer the questions
| Question | How to answer it from logs |
|---|---|
| Are important pages crawled? | List canonical URLs from the sitemap; join with the last Googlebot request date per URL |
| How long until new pages are crawled? | Compare publication time with the first verified Googlebot request |
| Where does crawling go? | Share of requests per section and per URL type |
| Is crawling wasted? | Share of requests to parameter URLs, redirects, 404s and duplicates |
| Are there errors? | 5xx and 429 responses to Googlebot, by URL and time |
| Did a change work? | Before and after comparison of the above |
Worked analysis with illustrative records
Here is a hypothetical seven-day sample of verified Googlebot smartphone requests on an online store:
| Segment | Requests | Share |
|---|---|---|
| Product pages (canonical) | 18,400 | 23% |
| Category pages | 6,200 | 8% |
| Filter parameter URLs | 41,600 | 52% |
| Redirecting URLs | 7,200 | 9% |
| 404 responses | 4,000 | 5% |
| 5xx responses | 2,600 | 3% |
| Total | 80,000 | 100% |
The figures are invented for the illustration, and the shares are computed from them: 41,600 / 80,000 is 52 percent.
What the sample suggests:
- More than half the crawling goes to filter combinations. If those URLs have no search value, they compete with products and categories for Google’s attention. This is the evidence that makes a crawl budget intervention worth considering.
- 9 percent goes to redirects. Internal links or sitemaps still point to old URLs.
- 3 percent receive server errors. The timestamps show the errors cluster during a nightly batch job. Google’s crawl budget guide says server errors and rate-limiting signals lower the crawl capacity limit (Google’s crawl budget guide).
- Product coverage: joining the product sitemap (40,000 URLs) with the request log shows 31,000 products not requested during the week. Not every product needs weekly crawling, but new products in that group need a discovery path.
Actions follow from the evidence: block crawling of low-value filter patterns, update internal links to final URLs, move the batch job or add capacity, and check that new products are linked from category pages.
Status codes to watch
- 200 for pages that should not exist (soft 404 candidates).
- 301/302 in volume (internal links and sitemaps pointing at old URLs; see redirect chains).
- 304 (Not Modified), which shows conditional requests working.
- 404/410 on URLs still linked internally.
- 429 and 5xx, which Google’s documentation says cause its crawlers to slow down; the 5xx diagnosis workflow separates their causes.
Tools and practicalities
- Logs can be large; analysis can be done with command-line tools, a database or a log analysis product. The method is the same.
- Keep at least several weeks of logs to see patterns, and store them in a way that permits joining with sitemap and crawl data.
- Strip or hash personal data in logs according to privacy obligations before sharing them with third parties.
- Make sure logs include the full request path and query string; some default configurations omit them.
Relationship to crawl budget decisions
Log analysis measures; it does not decide. Whether crawl patterns justify blocking sections, changing navigation or adding server capacity is a crawl budget decision, and for sites without the characteristics Google lists for crawl budget concerns, the answer may be to do nothing. Logs make that decision evidence-based instead of a guess.
Limits
- Logs show requests, not indexing decisions.
- They show what one server layer saw; a CDN may have answered some requests before they reached the origin.
- A user-agent label is not proof of identity until verified.
- A missing request in a week is not a problem by itself; crawl frequency varies by page importance and change rate.