Blocking domestic Google's in Robots.txt
-
Hey,
I want to block Google.co.uk from crawling a site but want Google.de to crawl it.
I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers?
any ideas?
Thanks
B
-
Thanks Guys for all of the help.
I think we will just implement cross domain GeoIP redirects to ensure users get the right location and currency.
Cheers
-
Are you having the issue of your .de pages ranking in .co.uk instead of your .co.uk pages?
If that's the case then I'd look towards usage of HREFLANG both on-page and in the xml sitemaps. That is going to provided Googlebot with a better view of the country-language targeting for the site.
-
Hi, country specific search engine spiders cannot be blocked using robots.txt file or any other method. However, you can block certain IP ranges pertaining to certain countries.
Best regards,
Devanur Rafi
-
Hi Gareth,
I don't think this is going to work as every crawler by Google is run by the same useragent: Googlebot. What you could do but what I really wouldn't recommend to do is generating the robots.txt automatically. Check if the IP address of the user is in another country and then Disallow them. It probably won't work as the crawler from let's say Germany could also be used for the UK.
Also the specific data for a countries search engine gets collected first and then gets looked at to see what they need users to serve, not the other way around that content gets acquired for a specific country.
Hope this helps!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Search Console International Targeting - Works for Hungary, but not Ireland - Why?
company.com (root)
International SEO | | scottclark
USA - lang="en" | GSC target: USA (shows US site in SERPs for "companyname" search)
company.com/hungary
Hungary - lang="hu" | GSC target: Hungary (shows Hungarian site in SERPs for "companyname" search)
AWESOMENESS company.com/ireland
Ireland - lang="en" | GSC target: Ireland (shows US site (doh!) in SERPs "companyname" search)
NOT RIGHT! It is our theory [please weigh in!] that because we don't have a company.com/usa folder, the TLD targeting (EN) is overriding other English language sites in some manner. In other words, the reason it's not overriding Hungary is because it's a different language. What must we do to get the Irish site ranked for "companyname" searches and to show by default in Ireland?0 -
What's the best homepage experince for an international site?
Greeting Mozzers. I have a question for the community, which I would appreciate your input on. If you have a single gTLD that services multiple countires, what do you think is the best homepage UX for the root homepage and why? So the example would be you own website www.company.org and target content to Germany, Japan and Australia with content through the folder structure eg. www.company.org/de-de If someone comes to the www.company.org from a region, would you: Redirect them based on location IP – so if from Germany they land on www.company.org/de-de Let them land on the homepage which offers location selection Let them land on a page with content and offer location selection eg. pop-up or obvious selection box Something I’ve not thought of… I'd appreciate your input. Thanks
International SEO | | RobertChapman0 -
Trying To Use Parent Company's Content In Another Country
Hi, I'm hoping someone can help me out here but this is what I am dealing with: Say John Smith Companies sells Widgets across the United States. They have also formed a company called "Widgets of Canada" in an effort to sell their Blue Widgets only in Canada and I am in responsible for their website. Recently, John Smith Companies completely redesigned their website and it now has a really slick look and is loaded with great widgets content. I would like to take their site and re-purpose it for use in Canada. However, I am concerned about duplicate content. I would be converting all the widget specifications from imperial to metric units, changing the title and description elements and also using a much different folders/ paths. Is this enough to avoid any issues with similar page content? Is there anything I can do with hreflang? Thanks
International SEO | | DohenyDrones0 -
International Sites - Sitemaps, Robots & Geolocating in WMT
Hi Guys, I have a site that has now been launched in the US having originally just been UK. In order to accommodate this, the website has been set-up using directories for each country. Example: domain.com/en-gb domain.com/en-us As the site was originally set-up for UK, the sitemap, robots file & Webmaster Tools account were added to the main domain. Example: domain.com/sitemap.xml domain.com/robots.txt The question is does this now need changing to make it specific for each country. Example: The sitemap and robots.txt for the UK would move to: domain.com/en-gb/sitemap.xml domain.com/en-gb/robots.txt and the US would have its own separate sitemap and robots.txt. Example : domain.com/en-us/sitemap.xml domain.com/en-us/robots.txt Also in order to Geolocate this in WMT would this need to be done for each directory version instead of the main domain? Currently the WMT account for the UK site is verified at www.domain.com, would this need reverifying at domain.com/en-gb? Any help would be appreciated! Thanks!
International SEO | | CarlWint0 -
Ranking UK company in Google.com
Hi all, I have a UK client with a .com domain, hosted on a US server, but the physical business premises is based in the UK. Their product is a really great product and available for export to the US. I want to rank them higher in the US, more specifically Google.com. I've helped them rank very well organically in the UK (google.co.uk) for some great terms, however they rank almost nowhere in google.com (gl=us) for the same terms, for example: In Google.co.uk they rank #3 for the key-phrase.
International SEO | | seowoody
In Google.com they rank #90 for the same key-phrase. I've got them some great US focused links with PR coverage including MSN Cars, nydailynews.com etc. I just wondered if there was any one "golden ticket" for boosting US rankings? I've read that a physical business premises located in the US helps a lot. Can anyone confirm this and if so, would a rented PO box in the US help? The site has great social signals too, growing twitter following and many FB likes/shares etc. Any other tips/advice? Thanks in advance,
Cheers,
Woody 🙂0 -
Does Google take into account the place where the server is hosted to rank the pages.
Does Google take into account the place where the server is hosted to rank the pages. What I mean is, if I have a server in USA and I am working for the Spain marketplace: Will Google rank better my pages for this market if the server were hosted in Spain?
International SEO | | NorbertoMM0 -
Risks of Migrating tld's to sub folders
Hi Guys, I am thinking of migrating our .co.nz and our .co.uk websites into sub folders on our .com website (eg: .com/uk and .com/nz). Do you think this is a risky strategy in regards to our performance in the localised search engines or should the centralisation of all these websites and their link authority into the .com help us move up the rankings? We are thinking of doing this in the next week, we have some really good rankings for the local googles, however we also have plenty of phrases sitting just on page 2 and I was hoping this might help boost them onto page 1? Has anyone else had experience migrating tld sites to sub folders on a .com and if so what was your experience of the impact on search rankings in the local googles and the timeframe that these changes took to have an effect? Did you have any negative results?
International SEO | | ConradC0 -
Migration from tld's to .com sub folders
Hi Guys, We currently operate five websites, 1 on .co.uk, 1 on .co.nz, 1 on .de and 1 on .com (geo targeted to USA) and 1 on .com/au (targeted at Australia). Open Site Explorer currently credits our .co.uk with 212 unique domains linking to us, our .com has 130, our .co.nz has 110 and our .de (which is new) has around 10. We have a website on .com/au targeting Australia and we have gained around 30 - 40 links into this sub folder. Our rankings in google australia for this website are fantastic and it would appear to me that we have inherited all the domain authority of our .com. The UK is currently our most important market and we operate a website on a .co.uk there. Our main competitors there have around 300 - 400 unique domains linking to them. What I am thinking of doing is deploying our UK content onto our .com root domain (which is currently geo targeted at the US which is a really small market for us) and redirecting all of the .co.uk pages at the root folder of the .com and changing the geo targeting of the .com to the UK. Additionally I was going to migrate our .co.nz and our .de websites into .com/nz/ and .com/de/ sub folders. I will also create a new .com/us/ folder for the US. I can only go off the fact that the only sub folder website we have (.com/au) has been very successful for us. Do you think migration of all of these websites onto the .com domain using sub folders will provide a meaningful boost to our rankings by virtue of having more back links into one domain? Are there any big risks in doing so and how long would you expect the redirects and changes to be picked up by google. I really appreciate any help and comments on this. Kind Regards
International SEO | | ConradC
Conrad Cranfield0