Blocking domestic Google's in Robots.txt
-
Hey,
I want to block Google.co.uk from crawling a site but want Google.de to crawl it.
I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers?
any ideas?
Thanks
B
-
Thanks Guys for all of the help.
I think we will just implement cross domain GeoIP redirects to ensure users get the right location and currency.
Cheers
-
Are you having the issue of your .de pages ranking in .co.uk instead of your .co.uk pages?
If that's the case then I'd look towards usage of HREFLANG both on-page and in the xml sitemaps. That is going to provided Googlebot with a better view of the country-language targeting for the site.
-
Hi, country specific search engine spiders cannot be blocked using robots.txt file or any other method. However, you can block certain IP ranges pertaining to certain countries.
Best regards,
Devanur Rafi
-
Hi Gareth,
I don't think this is going to work as every crawler by Google is run by the same useragent: Googlebot. What you could do but what I really wouldn't recommend to do is generating the robots.txt automatically. Check if the IP address of the user is in another country and then Disallow them. It probably won't work as the crawler from let's say Germany could also be used for the UK.
Also the specific data for a countries search engine gets collected first and then gets looked at to see what they need users to serve, not the other way around that content gets acquired for a specific country.
Hope this helps!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My Website Not Showing In Google English Search Results
My website is not visible on Google English. Selecting the language of Google in Hindi, Spanish, etc., my pages are visible in search results.
International SEO | | Jude_Wix0 -
How to check Robots.txt File
I want to know that how we are going to check that the Robots.txt File of the website is working properly. Kindly elaborate the mechanism for it please.
International SEO | | seobac1 -
In the U.S., how can I stop the European version of my site from outranking the U.S. version?
I've got a site with two versions – a U.S. version and a European version. Users are directed to the appropriate version through a landing page that asks where they're located; both sites are on the same domain, except one is .com/us and the other is .com/eu. My issue is that for some keywords, the European version is outranking the U.S. version in Google's U.S. SERPs. Not only that, but when Google displays sitelinks in the U.S. SERPs, it's a combination of pages on the European site and the U.S. site. Does anyone know how I can stop the European site from outranking the U.S. site in the U.S.? Or how I can get Google to only display sitelinks for pages on the U.S. site in the U.S. SERPs? Thanks in advance for any light you can shed on this topic!
International SEO | | matt-145670 -
How can I change the currency Google lists my products with in the SERP?
I.e. This product - http://www.absoluteautomation.ca/fgd400-sensaphone400-p/fgd400.htm - shows up as USD in the SERP. (In the US it just won't show a currency, if Canada it will show USD on the SERP). My pricing is all in CAD, how can I tell Google this? (It knows pricing is CAD in my Google Product Listings/Merchant Center). Thanks!
International SEO | | absoauto0 -
What is the best way to generate an automatic sitemap for google, bing and yahoo?
Hi guys, I have 3 international sites So far I recieved a Sitemap that was generated by an seo company to use and submit to google, for our co.nz domain, I have been told to submit this also to bing and yahoo. Can anyone tell me if I can submit the same sitemap for the com.au and com or would i need to generate a new sitemap for each domain? I have been told, everytime we change content we have to keep submitting, is there a way to do this autmatically as we will be writting alot of content daily. Any recommendations or suggestions?
International SEO | | edward-may0 -
Google Analytics & Webmaster Tools Filtering
I've just set up a client who uses internationalization on Google Webmaster tools and Google Analytics. For easier management they opted to use subfolders rather than subdomains or cctlds. So I set up a Google Analytic Property, with one Unfiltered profile, and another 3 profiles filtered per language. With the main language English being exclude anything starting /fr/ /de/ as it resides on root. The filters seem to work fine; however after linking this to the Google Webmaster Account to be able to access Search Engine Optimization I do not seem to get any language filtered data. I was wondering if someone had any idea or possible solution to this problem. As I would expect to at least have the Landing Pages if not exactly the keywords filtered by the same criteria that the rest of the data is being filtered. I know there's also an option to create a separate webmaster tools account, however this way I still cannot filter just the English; and I cannot link it to all the separate profiles.
International SEO | | jonmifsud0 -
Google suggesting a translation
Hi everybody, I notice since some months that Google when used for german language results proposes a translation next to the listing of one of my websites. When searching for english results (hl=en) it does not propose a translation! My website is clearly in german (given as target in GWMT and by meta tag). Other pages on the same domain are not treated this way by Google. No translation is proposed for all subpages of this website. Obviously, Google considers the homepage of this website english instead of german. Any fix for that? It is a *.org Wolfgang
International SEO | | wgr_strategic0 -
Hotel Multi country targeting + Google Local + TLDs => "MesSEO"
Hi guys, I own a guesthouse which is facing a messy structural problem in its own web presence: Portuguese: www.residencia-aeminium-coimbra.com.pt English: www.residencia-aeminium-coimbra.com Spanish: www.residencia-aeminium-coimbra.com/espanol Looking for success in the long term, a few years ago, we decided to host 3 TLDs: 1 for global international english, 1 for local portuguese and 1 for the main foreign market Spain (we already redirected it to a subfolder in the .com in the meanwhile). We tried to promote each one of these in their targeted markets - mainly the .com got back-links links and authority. With time, google local appeared and changed the face of google accommodation rankings. Google local are now the top results for most searches, even outside the country, and I came to the conclusion that the TLD assets were now a problem. For instance, I can only add 1 domain to google local..what language should this be? It is the same for most social media presence..How should international markets be targeted in a local page? Since TLDs do not seem to be an advantage right now, I am thinking we should be moving to a single domain and use a folder structure so we can use it everywhere. Questions: 1. In terms of structure (TLD or Folder) and multi-lingual targeting what is the current best practice for hotels that show in local results but promote internationally? 2. What language should I point google local results to? And our Facebook page, etc? 3. If I move things around to a folder structure, what domain should I use? the .com is in english and has the most authority and links according to opensiteexplorer the .com.pt is in portuguese, our local language and our main market (but only 35% share) should I create a new domain ".pt"? 4. I don't think that geo-targeting the languages is worth it in this case, what do you think? Kind Regards
International SEO | | retsimister
Ricardo Madeira
Residencia Aeminium Coimbra0