Can you confirm legitimate Google Bot traffic?
-
We use Cloudflare as a firewall. I noticed a significant number of blocks of bot traffic. One of the things they do is try to block bad bot traffic. But it seems they are mistakenly blocking Google Bot traffic. If you use Cloudflare, you may want to look into this as well. Also, can you confirm if the following IPs are for legitimate Google Bots?
66.249.79.88
66.249.79.65
66.249.79.8066.249.79.76
Thanks,
-
Thank you very much for all of your feedback folks. I really appreciate it. Cloudflare has acknowledged that they are aware of the issue and working to correct it. It also affected Bing Bots. I ended up white listing the IP ranges, and our Google crawl has increased as a result.
-
Dammit, my mistake.
Was proabaly still asleep. I've checked with my log files and updated itThanks Alex!
-
10.0.0.0/8 is a private subnet and therefore unroutable, so I don't think you would have seen traffic from that subnet from Googlebot!
-
What Alex has pointed out is what you're needing.
Also, I'd like to remember you that Googlebot crawls from several IP ranges. Not only the 66.xxx.xxx.xx.
EDIT:
Alex pointed me out that I've made a mistake. So I've checked my data in server logs.
In some of the sites I'm working, most of the traffic comes from the range: [66.249.6x.xxx-66.249.7x.xxx]Always, when in doubt, with a reverse DNSlookup you'll find out whether it's an official GoogleBot or not.
Hope it helps.
Best luck.
Gaston -
Following the advice here: https://support.google.com/webmasters/answer/80553?hl=en I can confirm that they are indeed Googlebot.
If you use Fetch as Google in Search Console, are you seeing errors?
Cloudflare has an article here on the subject:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Plus Places Error
We have a large amount of clients and when we are updating their Google Plus Places listing, the ad is still presented as active, however it is not live on Google and when you click to view the listing we get this message 'We currently do not support this location' I have researched this and found many people are having this issue, but no solutions as of yet. Can anyone shed some light on to this because some of our clients are not thrilled at the moment. Thanks Jon
Technical SEO | | Jon_bangonline0 -
Google is indexing my directories
I'm sure this has been asked before, but I was looking at all of Google's results for my site and I found dozens of results for directories such as: Index of /scouting/blog/wp-includes/js/swfupload/plugins Obviously I don't want those indexed. How do I prevent Google from indexing those? Also, it only seems to be doing it with Wordpress, not any of the directories on my main site. (We have a wordpress blog, which is only a portion of the site)
Technical SEO | | UnderRugSwept0 -
Google Webmaster Sitemap *pending*
Hey guys, I've noticed that my sitemap has been "pending" for quite some time in Google Webmaster tools. This leads me to believe that Google is not indexing my site. Could someone help me and point me to what I'm doing wrong? My site is The Tech Block
Technical SEO | | ttb0 -
How to block google robots from a subdomain
I have a subdomain that lets me preview the changes I put on my site. The live site URL is www.site.com, working preview version is www.site.edit.com The contents on both are almost identical I want to block the preview version (www.site.edit.com) from Google Robots, so that they don't penalize me for duplicated content. Is it the right way to do it: User-Agent: * Disallow: .edit.com/*
Technical SEO | | Alexey_mindvalley0 -
Can anyone tell me where my site can be improved
My website is www.theradiatorgallery.com I have been trying to get on 1st page for designer radiators. I am currently ranked 21st just dropped to the 3rd page within the past hour, i was 20th before this. There are some sites that i would think i would rank better than due do i have better on page optimization, higher PA, DA and PR based on seo moz reports. Is there something wrong with my site that i am missing? the sites i feel i should rank better than are; radsnrails.co.uk, warmrooms.co.uk and a couple others Any help would be great! Thanks Guys
Technical SEO | | myloseo0 -
Google Pake Rank of Zero?
Hi! We've been running our website www.opzeggen.nl for a while now, but we still have a google page rank of 0. Searching for a keyword like 'opzeggen' does give us good ranking however. Should I be worried about this? And if so, is there a way to fix this? Best Regards, Pieter
Technical SEO | | greenonline0 -
Google Dmoz description in SERPS
My dmoz description is not as KW rich as my sites normal description. IS there an advantage or disadvantage to either? If so, How do I prevent google from doing this?
Technical SEO | | DavidS-2820610 -
Why Google did not index our domain?
Hi, We launched tmart 60 days ago and submitted to google, bing, yahoo 20 days later. But google had never indexed our website still when yahoo indexed it in one week. What we have checked or tried: 1. We got 20~50 inlinks in one month and now 81 inlinks via yahoo site explorer. 2. This domain has registered for 13 years and we purchased it from sedo last year. We
Technical SEO | | zt673
did not find any problems from domain archive pages. 3. Page similar: the homepage is 50% similar to one of our competitors when we just launched.
So we adjusted the page structure and modified the content one month later and decreased the similarity to 30% (by tools from webconfs.com) 4. Google Robots: googlebot crawled our website every day after we submitted for indexing.
We opened GWT account for it and added the xml sitemap last week. GWT said nothing
was wrong except the time of page loading. Our questions: Why google did not indexed our website? What should we do? Thanks, wu0