Blocking domestic Google's in Robots.txt
-
Hey,
I want to block Google.co.uk from crawling a site but want Google.de to crawl it.
I know how to configure the Robots.txt to block Google and other engines - is there a fix to block certain domestic crawlers?
any ideas?
Thanks
B
-
Thanks Guys for all of the help.
I think we will just implement cross domain GeoIP redirects to ensure users get the right location and currency.
Cheers
-
Are you having the issue of your .de pages ranking in .co.uk instead of your .co.uk pages?
If that's the case then I'd look towards usage of HREFLANG both on-page and in the xml sitemaps. That is going to provided Googlebot with a better view of the country-language targeting for the site.
-
Hi, country specific search engine spiders cannot be blocked using robots.txt file or any other method. However, you can block certain IP ranges pertaining to certain countries.
Best regards,
Devanur Rafi
-
Hi Gareth,
I don't think this is going to work as every crawler by Google is run by the same useragent: Googlebot. What you could do but what I really wouldn't recommend to do is generating the robots.txt automatically. Check if the IP address of the user is in another country and then Disallow them. It probably won't work as the crawler from let's say Germany could also be used for the UK.
Also the specific data for a countries search engine gets collected first and then gets looked at to see what they need users to serve, not the other way around that content gets acquired for a specific country.
Hope this helps!
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Problem to get multilingual posts indexed on Google
Last year on June I decided to make my site multi-lingual. The domain is: https://www.dailyblogprofits.com/ The main language English and I added Portuguese and a few posts on Spanish. What happened since then? I started losing traffic from Google and posts on Portuguese are not being indexed. I use WPML plugin to make it multi-lingual and I had Yoast installed. This week I uninstalled Yoast and when I type on google "site:site:dailyblogprofits.com/pt-br" I started seeing Google indexing images, but still not the missing posts. I have around 145 posts on Portuguese, but on Search Console it show only 57 hreflang tags. Any idea what is the problem? I'm willing to pay for an SEO Expert to resolve this problem to me.
International SEO | | Cleber0090 -
Google Search Console International Targeting - Works for Hungary, but not Ireland - Why?
company.com (root)
International SEO | | scottclark
USA - lang="en" | GSC target: USA (shows US site in SERPs for "companyname" search)
company.com/hungary
Hungary - lang="hu" | GSC target: Hungary (shows Hungarian site in SERPs for "companyname" search)
AWESOMENESS company.com/ireland
Ireland - lang="en" | GSC target: Ireland (shows US site (doh!) in SERPs "companyname" search)
NOT RIGHT! It is our theory [please weigh in!] that because we don't have a company.com/usa folder, the TLD targeting (EN) is overriding other English language sites in some manner. In other words, the reason it's not overriding Hungary is because it's a different language. What must we do to get the Irish site ranked for "companyname" searches and to show by default in Ireland?0 -
What language to use for URL's for Russian language?
Hi, Our site is in English, Spanish, Danish and Russian - the URL's are individual to the language they are in, but of course, Russian contains some strange characters so I decided not to use them in the URL's Any advice on how to create the URL's for russian language pages? thanks
International SEO | | bjs20100 -
My site is not showing on google.com ?
My website is not showing at all on google.com searches under search terms. It is showing if i search for my domain (xyz.com) but not by keywords. Searching by keywords and search terms shows my website on yahoo , bing and non US oogle as google.co.uk or google.com.pk on first page. Would appreciate any help in trying to understand why this is happening? The website is medicare.md. if you search for term "medicare doctors PG county maryland" it is #1 in bing and yahoo but not even showing on google.com first TEN pages, although not banned. Interestingly if you do that search on google.co.pk it is #4. Quite Puzzuling !! Would appreciate any help or advice . Sherif Hassan
International SEO | | sherohass0 -
Robots.txt issue with indexation
Hello i have a problem with one of the rules for robots.txt i have a multilingual mutation of entire page on www.example.com/en/ I want to make indexable /allow/ the main page under /en/ but not indexable /disallow/ everything else under /en/* Please help me how to write the rule.
International SEO | | profesia0 -
Google Webmaster Tools - Geographic target - Time till change
One of my clients is targeting UK customers. They have UK IP address. I then noticed that they had their Geographic target set to "Target users in: United States". I advised them to change this. Anyone know: How this could have been set, the client says they didn't do it? How long will it take for Google to see this has been changed? Thanks in advance. Justin
International SEO | | GrouchyKids0 -
How do you see Google results specific to location?
We run a Canadian website and are interested in seeing what SERPs look like from specific postal codes. Is there any way to manipulate Google to think our IP address comes from another location? Thanks!
International SEO | | ClaytonKendall0 -
How can I view Google.com SERPs from outside the US?
If I go to Google.com I get redirected back to Google.co.uk and search results have a UK bias. I'm trying to research the US market and have a hazy recolection of Rand demonstrating how you can add a few characters to the google.co.uk url to see US results - just can't remember what video I saw it in. Any ideas? Thanks a lot
International SEO | | trbaldwin0