Blocking domestic Google's in Robots.txt
-
Hey,
I want to block Google.co.uk from crawling a site but want Google.de to crawl it.
I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers?
any ideas?
Thanks
B
-
Thanks Guys for all of the help.
I think we will just implement cross domain GeoIP redirects to ensure users get the right location and currency.
Cheers
-
Are you having the issue of your .de pages ranking in .co.uk instead of your .co.uk pages?
If that's the case then I'd look towards usage of HREFLANG both on-page and in the xml sitemaps. That is going to provided Googlebot with a better view of the country-language targeting for the site.
-
Hi, country specific search engine spiders cannot be blocked using robots.txt file or any other method. However, you can block certain IP ranges pertaining to certain countries.
Best regards,
Devanur Rafi
-
Hi Gareth,
I don't think this is going to work as every crawler by Google is run by the same useragent: Googlebot. What you could do but what I really wouldn't recommend to do is generating the robots.txt automatically. Check if the IP address of the user is in another country and then Disallow them. It probably won't work as the crawler from let's say Germany could also be used for the UK.
Also the specific data for a countries search engine gets collected first and then gets looked at to see what they need users to serve, not the other way around that content gets acquired for a specific country.
Hope this helps!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does not having any hreflang tags for U.S Visitors lead to an increase in International Visitors?
I have seen a massive increase in International Visitors on our website and visitors within the United States dropped off hard this month (by about 20%). Could it be possible that not having any hreflang tags can lead to an increase in International Customers visiting the site even though your sitemap is set to "Target users in United States" within the Google Search Console? In the Google Search Console, I have International Targeting set to "Target users in United States." However, Google Search Console is saying our site doesn't have any hreflang tags. In the Google Search Console, it says "Your site has no hreflang tags. Google uses hreflang tags to match the user's language preference to the right variation of your pages." I'm not sure when that was flagged, but recently we have seen a massive increase in International Visitors to our site from countries such as Russia, Vietnam, Indonesia, the United Kingdom and so on. This poses a problem since our chances of turning one of those visitors into a customer is extremely slim. Along with that, nearly every international customer is contributing to an extremely high Bounce Rate. Attached is a screenshot of the Error about hreflang tags. https://imgur.com/a/XZI45Pw And here is a screenshot of the Country we are targeting. https://imgur.com/a/ArpWe9Z Lastly, attached is a screenshot of all of the Countries that visited our site today: https://imgur.com/a/d0tNwkI
International SEO | | MichaelAtMSP1 -
What's the best homepage experince for an international site?
Greeting Mozzers. I have a question for the community, which I would appreciate your input on. If you have a single gTLD that services multiple countires, what do you think is the best homepage UX for the root homepage and why? So the example would be you own website www.company.org and target content to Germany, Japan and Australia with content through the folder structure eg. www.company.org/de-de If someone comes to the www.company.org from a region, would you: Redirect them based on location IP – so if from Germany they land on www.company.org/de-de Let them land on the homepage which offers location selection Let them land on a page with content and offer location selection eg. pop-up or obvious selection box Something I’ve not thought of… I'd appreciate your input. Thanks
International SEO | | RobertChapman0 -
Trying To Use Parent Company's Content In Another Country
Hi, I'm hoping someone can help me out here but this is what I am dealing with: Say John Smith Companies sells Widgets across the United States. They have also formed a company called "Widgets of Canada" in an effort to sell their Blue Widgets only in Canada and I am in responsible for their website. Recently, John Smith Companies completely redesigned their website and it now has a really slick look and is loaded with great widgets content. I would like to take their site and re-purpose it for use in Canada. However, I am concerned about duplicate content. I would be converting all the widget specifications from imperial to metric units, changing the title and description elements and also using a much different folders/ paths. Is this enough to avoid any issues with similar page content? Is there anything I can do with hreflang? Thanks
International SEO | | DohenyDrones0 -
In the U.S., how can I stop the European version of my site from outranking the U.S. version?
I've got a site with two versions – a U.S. version and a European version. Users are directed to the appropriate version through a landing page that asks where they're located; both sites are on the same domain, except one is .com/us and the other is .com/eu. My issue is that for some keywords, the European version is outranking the U.S. version in Google's U.S. SERPs. Not only that, but when Google displays sitelinks in the U.S. SERPs, it's a combination of pages on the European site and the U.S. site. Does anyone know how I can stop the European site from outranking the U.S. site in the U.S.? Or how I can get Google to only display sitelinks for pages on the U.S. site in the U.S. SERPs? Thanks in advance for any light you can shed on this topic!
International SEO | | matt-145670 -
If domain mapping subfolders to TLD's is it perceived as a fully separate entity/site therafter ?
Hi I take it once you have domain mapped a country specific subfolder to a country specific TLD (for better local region targeting reasons) Google perceives it as a completely separate entity and it no longer shares any of the parent sites domain benefits (such as domain authority etc) so from that point on requires its own dedicated link building etc ? All Best Dan
International SEO | | Dan-Lawrence0 -
Dynamic Google search snippet text based on user's language
On Google search results page, I want to show search snippet text (of my webpage) in Hindi language if user is user is using Google in Hindi language. If user chose another language on Google search page, my snippet text should be shown in that language. Is this possible? How?
International SEO | | Avinashmb0 -
Google suggesting a translation
Hi everybody, I notice since some months that Google when used for german language results proposes a translation next to the listing of one of my websites. When searching for english results (hl=en) it does not propose a translation! My website is clearly in german (given as target in GWMT and by meta tag). Other pages on the same domain are not treated this way by Google. No translation is proposed for all subpages of this website. Obviously, Google considers the homepage of this website english instead of german. Any fix for that? It is a *.org Wolfgang
International SEO | | wgr_strategic0 -
Geo Targeting for Similar Sites to Specific Countries in Google's Index
I was hoping Webmaster Tools geo targeting would prevent this - I'm seeing in select google searches several pages indexed from our Australian website. Both sites have unique TLDs: barraguard.com barraguard.com.au I've attached a screenshot as an example. The sites are both hosted here in the U.S. at our data center. Are there any other methods for preventing Google and other search engines from indexing the barraguard.com.au pages in searches that take place in the U.S.? dSzoh.jpg
International SEO | | longbeachjamie0