What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?
-
Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?
-
Hi Keith,
In my experience Google mainly crawls from the US.
You're quite right, GeoIP redirection can cause major issues with indexation - as if you're redirecting everything from the US to the .com googlebot can't see the .co.uk site.
As such, I'm not a fan. Rather than implementing a hard redirect I prefer Amazon's approach. If you visit the amazon.com site from a UK IP you get a javascript overlay that invites you to visit the .co.uk version of the site instead - they let the user decide which site to view rather than actually redirecting them.
This is a nice solution as it ensures that the search bots can crawl both versions of the site, and rankings aren't endangered.
I hope this helps,
Hannah
-
Keith, I am having the same issue and I agree with you. The fact that Google has data centers in Europe does not necessarily mean the algos are indexing from there. I also want to set Europe and US GeoIP redirection. It would be great to get Mozers opinions on this. Hopefully this post gets freshly reviewed
-
Interesting question - I'd quite like to know what happens here too.
Matt Cutts recently posted a video on cloaking (http://youtu.be/QHtnfOgp65Q) saying that as long as you don't do anything 'special' for Googlebot you're OK, but presumably if you are redirecting IP's based on location and you don't want to prevent Googlebot form accessing your site then effectively you have to do something 'special' for Googlebot (e.g. you're doing one thing for everyone else and a different thing for Googlebot).
-
Hi,
The data center locations is interesting, but it isn't what I was looking for. I need to know whether Google crawls the web from any IP other than US IPs.
To clear up the second question, let me be more specific:
Let's say Google is crawling a .co.uk site from a US IP address. The site is using GEO IP redirection to redirect all US traffic to the .com site. Therefore, when Google attempts to crawl the .co.uk from the US IP address site it will be redirected to the .com site, never seeing the .co.uk site. Can anyone confirm that this is what happens?
-
I found an article from 2008 that shows Google data centre locations: http://bit.ly/mONhf9
Your other question is a bit confusing. Why Google wouldn't see the UK site?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
US site vs New Canadian site for Brand
Hi Everyone, My company decided to create a Canadian site for Canadian customers. How do I slowly transition the US site for ranking in Google.ca? I was thinking of using robots.txt to block Google.Ca from crawling the US site? Can anyone provide some advice oh how this should be managed? Thank you!
International SEO | | JMSCC0 -
Soft Launch App with Country Targeting To Match?
Hi,
International SEO | | eTinaRose
We're gearing up to do a soft launch of our app in a few countries, such as Australia. Our website says to request early access since the app is not yet publicly available in the apple store. However, it will be in those soft launch countries. This is what I'm considering... Creating a subdomain site such as au.liquidtext.net (liquidtext.au is not available). And setting that site, au.liquidtext.net, to be country targeted to Australia via webmaster tools. Once the app is available internationally, and the main site has been updated to show this, I would then redirect any subdomains to the main site. Thoughts? Are there any negative repercussions to this this approach that I am not considering? Thank you for your help Moz community!! Tina0 -
Help! A never before asked query about using a ccTLD but hosting in a different country
Hi Guys, I've a website that has a India specific domain ending with .org.in. The website has ALL the traffic from India (as mentioned earlier, it's a website meant only for audience from India). Currently this _.org.in_domainis hosted on a server located in India. I'm thinking of hosting this website in Singapore. Do you think that will negatively affect the current rankings of the website (i.e. changing the server location of my website from India to Singapore)?
International SEO | | seotoseo0 -
Keyphrase ranking a geo-redirected site in Google
Hi all This is the situation. I have a client who runs a number of ccTLD sites (all exact match brand name domains), including a .com which they use for the US. This is a hair care product and due to Advertising Standards Authority (UK) restrictions, they cannot use a certain phrase to promote their products - 'hair loss' on the domain.co.uk site. However, in the US, there is no such restriction and can use wording this on the site. A brand name search in google.co.uk brings up .co.uk as 1st result and .com as 2nd result, so the .com is indexed in google.co.uk. Any non-US user visiting domain.com will be redirected to their ccTLD site. Here's my question - could I feasibly get the domain.com site ranking in google.co.uk for certain 'hair loss' based keyphrases, considering the fact that I can mention it in the copy on there but not on the domain.co.uk site. Would I need to remove any Geographic Target in the WMT account for domain.com? Or is this a form of Google cloaking and could see the site penalised? Thanks
International SEO | | Coolpink0 -
How well does Google's crawlers understand foreign websites?
I speak 5 languages and therefore have the opportunity to do on-page SEO and content writing for 5 different cultures. This question to me has much to do with the way Google translator works. It doesn't, trust me! Which then makes me wonder how the web crawlers, which are designed with English in mind, can fairly and equally attribute the same ranking points to a foreign website. Since Google seems to use sematic search technology I'm wondering if foreign sites have it easier or not. Any ideas?
International SEO | | MassivePrime0 -
How can I replicate India SERPs from US
I was recently checking out our analytics keywords stats and I am seeing traffic noted as google organic traffic coming from India. When I do a search on http://www.google.co.in/ for those terms using a "incognito" browser with no cookies, and not logged in I don't see our site in the first 100 results. The same thing happens on google.com if I try to set the account in india. I am trying to see if this might be an issue with an adwords campaign not getting auto-tagged (auto-tagging is turned on in adwords), or if there is something else going on. The keyword campaign gets many more clicks than analytics shows for this organic option so perhaps it is an odd rare occurrence. I am glad to have the traffic, but I want to be sure it is actually tracking correctly as organic, or if PPC is not tagging for some reason 😉 Any suggestions or thoughts?
International SEO | | SL_SEM0 -
Does publishing your physical company address increase your Google Trust?
The question is in the title: Does publishing your physical company address increase your Google Trust? I.e. on the contact page of the website, providing a physical address. The address would match the WHOIS entry for the domain, and allow customers to contact you via post.
International SEO | | Peter2640 -
One global blog or a blog for each country?
We have a blog on each of our country sites, but the content on the English speaking sites is shared i.e. locally produced in US is mixed up with stuff produced in UK and vice versa. I'm not concerned about duplicate content because we have taken the necessary measures to let the search engines know that we have a US site for the US and and UK site for the UK. My question to you is whether we should develop the blog independently in each country or develop one blog to satisfy our global sites. I believe a local blog for each country is better because the content would speak to the audience better and any reference points or product and price points would be 100% relevant to the audience.
International SEO | | Red_Mud_Rookie0