What countries does Google crawl from? Is it only US or do they crawl from Europe and Asia, etc.?
-
Where does Google crawl the web from? Is it in the US only, or do they do it from a European base too? The reason for asking is for GeoIP redirection. For example, if a website is using GeoIP redirection to redirect all US traffic to a .com site and all EU traffic to a .co.uk site, will Google ever see the .co.uk site?
-
Hi Keith,
In my experience Google mainly crawls from the US.
You're quite right, GeoIP redirection can cause major issues with indexation - as if you're redirecting everything from the US to the .com googlebot can't see the .co.uk site.
As such, I'm not a fan. Rather than implementing a hard redirect I prefer Amazon's approach. If you visit the amazon.com site from a UK IP you get a javascript overlay that invites you to visit the .co.uk version of the site instead - they let the user decide which site to view rather than actually redirecting them.
This is a nice solution as it ensures that the search bots can crawl both versions of the site, and rankings aren't endangered.
I hope this helps,
Hannah
-
Keith, I am having the same issue and I agree with you. The fact that Google has data centers in Europe does not necessarily mean the algos are indexing from there. I also want to set Europe and US GeoIP redirection. It would be great to get Mozers opinions on this. Hopefully this post gets freshly reviewed
-
Interesting question - I'd quite like to know what happens here too.
Matt Cutts recently posted a video on cloaking (http://youtu.be/QHtnfOgp65Q) saying that as long as you don't do anything 'special' for Googlebot you're OK, but presumably if you are redirecting IP's based on location and you don't want to prevent Googlebot form accessing your site then effectively you have to do something 'special' for Googlebot (e.g. you're doing one thing for everyone else and a different thing for Googlebot).
-
Hi,
The data center locations is interesting, but it isn't what I was looking for. I need to know whether Google crawls the web from any IP other than US IPs.
To clear up the second question, let me be more specific:
Let's say Google is crawling a .co.uk site from a US IP address. The site is using GEO IP redirection to redirect all US traffic to the .com site. Therefore, when Google attempts to crawl the .co.uk from the US IP address site it will be redirected to the .com site, never seeing the .co.uk site. Can anyone confirm that this is what happens?
-
I found an article from 2008 that shows Google data centre locations: http://bit.ly/mONhf9
Your other question is a bit confusing. Why Google wouldn't see the UK site?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Wants to Rank Well in Multiple Countries
We have been using moz subscription from last few months & are quite happy so far. We just wanted to explore more out of it & wanted to understand how to approach the optimization for having us rank well in US. At the same time we don't wanted to lose upon what we have achieved till now. I have read through some blog posts & found that changing the international targeting in GWT can also hurt. My vision is to rank well in multiple countries but at the same time do not lose rankings in India. Any help here would be appreciated so whatever we do, we do it right.
International SEO | | fourseven0 -
Problems with the google cache version of different domains.
We have problems with the google cache version of different domains.
International SEO | | Humix
For the “.nl” domain we have an “.be” cache..
Enter “cache:www.dmlights.nl” in your browser to see this result. Following points are already adapted: Sitemap contains hreflang tag Sitemap is moved to the location www.dmlights.nl/sitemap.xml We checked the DNS configuration Changed the Content language in de response header to : Content-Language: nl-NL Removed the cache with webmastertools Resolved serverrequest errors. Can anyone provide a solution to fix this problem? Thanks, Pieter0 -
Google Webmaster showing error for [hreflang='x-default']
Hi There! Using [hreflang='x-default'] tag to target language specific countries on our site but Google Webmaster showing errors even implementation made as per Google guideline but one thing is not clear and we are not sure, this is the reason behind it. Error is showing up only on those pages where 'Google Parameters' are used. For example : https://www.sitegeek.com/a2hosting?grank=open 'grank=' is defined as a 'Google Parameters' and on the above page 'hreflang' tags are : Also, on page https://www.sitegeek.com/a2hosting [without Google Parameters] same above 'hreflang' tags are taken. But, There is no error on second page URL where no 'Google Parameters' in URL. Therefore, error showing on first URL where 'Google Parameters' are taken. Is this the issue or not? suggest how to remove? -- Rajiv S9vhl3T
International SEO | | gamesecure0 -
How to Use Additional Country Domains
Greetings Mozlings We have a .co.uk address but mostly sell to South East Asia. Is there a benefit to getting .com.my, .com.sg, .co.id addresses If we do how should we use them - a simple redirect to the .co.uk? A single page site with a link to the .co.uk Some other way We're resource and time lite so we're looking to maximise the benefits with the minimum time investment Cheers Denis
International SEO | | Zippy-Bungle0 -
Blocking domestic Google's in Robots.txt
Hey, I want to block Google.co.uk from crawling a site but want Google.de to crawl it. I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers? any ideas? Thanks B
International SEO | | Bush_JSM0 -
How to fix the duplicate content problem on different domains (.nl /.be) of your brand's websites in multiple countries?
Dear all, what is the best way to fix the duplicate content problem on different domains (.nl /.be) of your brand's websites in multiple countries? What must I add to my code of websites my .nl domain to avoid duplicate content and to keep the .nl website out of google.be, but still well-indexed in google.nl? What must I add to my code of websites my .be domain to avoid duplicate content and to keep the .nl website out of google.be, but still well-indexed in google.nl? Thanks in advance!
International SEO | | HMK-NL3 -
Same website in different countries, best practices for SEO?
Hey Guys, I have read several similar questions regarding mine, but none seem to truly cover my question. Basically, we have a company named Junair. We created the website for the company here in Australia (http://www.junair.com.au). As can be seen throughout the page, it mentions that it caters for both Australia and NZ (NZ has its own phone number). It does ok in the rankings at the moment, but rankings will continue to rise in the future once more links are getting picked up. Now however, the Junair team in NZ purchased the NZ domain http://www.junair.co.nz and redirected it to the Australian page. No matter which page you visit on the NZ URL, the URL will never change, and neither will the page title. They have now contacted us and asked to perform SEO on the NZ domain so the NZ domain would show up in searches on Google NZ. At the moment, when searching for "Junair" on google.co.nz, the Australian domain is coming up. How could I change this so the NZ URL would show instead? And what would be the best practices to perform SEO on the NZ URL, should I just create links pointing to http://www.junair.co.nz ? Thank you, Roderic
International SEO | | Michael-Goode0 -
Webmaster Tools crawl error shows Porn from China.
I have an ongoing issue with porn showing up in my crawl error report from Google Webmaster Tools. They show up like so… _ ads/adtrack.asp?id=monsterselfemp&type=2&redir=//deuxmalio.cn/m.php/BLACK-GRANNY-VIDS-612_ Domain name not found 0008-05-11 I have no idea what deuxmalio.cn is, and it doesn't show up in search. Would this negatively impact my rankings and what can I do about this? I receive A LOT of these, most of which I cannot repeat in this forum.
International SEO | | BeTheBoss0