Googlebot log file analyzer: verify real Googlebot visits
Search Console shows a summary of crawling; your server logs show every request. The analyzer pulls out Google's crawlers and separates real ones from fakes.
Your data stays in your browser. Nothing is uploaded or stored.
Key facts
- Google recommends verifying Googlebot with a reverse DNS lookup (the host name must be in googlebot.com, google.com or googleusercontent.com) followed by a forward DNS lookup that returns the same IP, or by matching the IP against Google's published IP ranges.
- Googlebot Smartphone and Googlebot Desktop obey the same user-agent token in robots.txt, and most Googlebot requests come from the mobile crawler.
- For most sites, Googlebot shouldn't request pages more often than once every few seconds on average.
- A user-agent string alone can be faked, which is why the IP check matters.
Download the checklist: diagnosing indexing problems
Step by step: status in Search Console, page availability, robots.txt, noindex, canonical URL, redirects, sitemap and backlinks, with what to do in each case.
Download PDF PDF · 2 pagesHow to use it
- Download access logs from your server or hosting panel (Apache or Nginx, common or combined format).
- Choose one or more log files, or paste log lines.
- Click Analyze log.
- Look at the error table first: these are URLs where Google got 4xx or 5xx responses.
- Check the “not in Google ranges” table for bots pretending to be Googlebot, and download the CSV for details.
Where to get the data
Apache or Nginx access logs
Usually access.log, in the combined format: IP, date, request, status, size, referrer and user agent.
Hosting control panels
Most panels offer raw access logs for download. Use raw logs, not statistics summaries.
CDN logs
If your CDN can export logs in common or combined format, they work too. JSON log formats aren't supported.
Why verify the IP
Anyone can send requests with a Googlebot user agent. A request is genuine only if its IP is in Google's published ranges, or if its reverse DNS name is in Google's domains and a forward lookup of that name returns the same IP. The tool checks IPs against the list Google publishes for its common crawlers.
What to look for
Errors on important URLs, many hits on parameter URLs or redirects, and days with no Google visits at all. Compare the most crawled URLs with the pages you actually want indexed.
Smartphone and desktop
The crawler type is read from the user agent: Googlebot/2.1 with a mobile browser string is Googlebot Smartphone, without it Googlebot Desktop. Other tokens, such as Googlebot-Image or Google-InspectionTool, are shown separately.
Next step
Fixed the errors? get the updated pages recrawled so Google sees the changes sooner.
Questions and answers
Is my log uploaded?
No. The file is read in your browser; only Google's public IP list is loaded from this site.
Which log formats work?
Apache and Nginx common and combined formats, which most servers use by default.
How current is the IP list?
It's copied from Google's published file each time this site is updated. If it can't be loaded, the tool says so and skips the check.
How big can the log be?
That depends on your device's memory. For very large logs, analyze one day or one file at a time.
Sources
- Verifying Googlebot and other Google crawlers — Google Search Central: https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot
- Googlebot — Google Search Central: https://developers.google.com/search/docs/crawling-indexing/googlebot
- Google's common crawlers — Google Search Central: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers