Duplicate site (disaster recovery) being crawled and creating two indexed search results
-
I have a primary domain, toptable.co.uk, and a disaster recovery site for this primary domain named uk-www.gtm.opentable.com. In the event of a disaster, toptable.co.uk would get CNAMEd (DNS alias) to the .gtm site. Naturally the .gtm disaster recover domian is an exact match to the toptable.co.uk domain.
Unfortunately, Google has crawled the uk-www.gtm.opentable site, and it's showing up in search results. In most cases the gtm urls don't get redirected to toptable they actually appear as an entirely separate domain to the user. The strong feeling is that this duplicate content is hurting toptable.co.uk, especially as .gtm.ot is part of the .opentable.com domain which has significant authority. So we need a way of stopping Google from crawling gtm.
There seem to be two potential fixes. Which is best for this case?
- use the robots.txt to block Google from crawling the .gtm site
2) canonicalize the the gtm urls to toptable.co.uk
In general Google seems to recommend a canonical change but in this special case it seems robot.txt change could be best.
Thanks in advance to the SEOmoz community!
-
It's a little tricky. While Andrea is right about Robots.txt - it's not great for removal once pages/domains are indexed, you can block the sub-domain with robots.txt and then request removal in Google Webmaster Tools (you need to create a separate account for the sub-domain itself). That's often the fastest way to remove something from the index, and if it has no search value, I might go that route. Just proceed with caution - it's a delicate procedure.
Doing 1-to-1 canonicalization or adding 301 redirects may be the next strongest signal (NOINDEX is a bit weaker, IMO). However, Google will have to re-crawl the sub-domain to do that, so you'll need to keep the paths open.
-
First, if the pages are already indexed then a robots.txt won't make them go away. A meta tag no index on the pages is the better solution. This allows search engines to "read" you page, see the no index tag and then work to remove the pages from index. A robots.txt doesn't necessarily accomplish the same result.
-
If you can do a 1-to-1 page canonicalization (each page on .co.uk is canonicaled to the equivalent page on the .com) then I would do that.
Otherwise, I would noindex the backup site.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Where to point canonical for m-dot site in the wake of Mobile-First Indexing
My client currently use an m-dot URL for their mobile site and while conducting a technical audit for their web properties, we have noticed that their desktop is using a self-referencing rel="canonical" while their mobile m-dot has no rel="canonical" tags. While our initial recommendation is to point the mobile m-dot point to the desktop using a rel="canonical" and the desktop point to the mobile using a rel="alternative," there have been hesitations about mobile first indexing and canonical tags. If Google will use the m-dot for indexing purposes moving forward, is the progressive recommendation to have the desktop point to the m-dot using a rel="canonical" and the m-dot point to the desktop using a rel="alternative" or to maintain the initially stated recommendation?
Intermediate & Advanced SEO | | Derek_Hawk0 -
Recovery from manual penalty, several sites sell same products
We are still struggling with the consequences of a manual penalty. Here the history we got a manual penalty in September 2013 on our main site kinderwagen.com for unnatural links we cleaned up our link profile and the penalty was lifted in February 2014 we did nothing in terms of link building until December 2014 from December 2014 we started to build natural links, we mostly gave away our products for reviews, that way we built first one content link per month now 2-3 content links per months we expanded our social acitivities on facebook and google plus we started a blog in the beginning of this year and rewrote our guide section For now the results of all this have been very modest. We are basically still stuck where we were after the penalty. In contrast our sites in other languages where we do similar activities perform quite well. There is one difference in Germany (where this kinderwagen.com is) we have other niche sites which partially sell the the same products. We made sure that every product is indexed only once, however it might still hurt the main site. Does anyone have experience in a similar situation or advice what we could do? Thanks in advance. Dieter
Intermediate & Advanced SEO | | Storesco0 -
URL Spoof Issue in Search Results
Hello! We could use some assistance diagnosing an issue. In order to avoid asking a convoluted question, I will try to break it down below: 1. A random foreign site is hacked and a subdirectory is added that is completely irrelevant to the root. a). i.e. http://www.um.org/prom_dresses/ 2. http://www.um.org/prom_dresses/ is just a phishing prom dress page 3. When you search "prom dress shop", the website that used to rank first (for good reason) was www.promdressshop.com. 4. www.promdressshop.com's home page has now been replaced by: um.org/prom_dresses/ – who is using prom dress shop's title tag and meta description. How is it possible that this hacked page (on um.org) is not only ranking above us, but is also starting to replace www.promdressshop.com's pages in search results. We do not believe www.promdressshop.com has been hacked but are open to any ideas. Please let me know if you would like any additional info. Thanks in advance! new
Intermediate & Advanced SEO | | LogicalMediaGroup0 -
Creating 20+ websites with links back to central site
Hey guys, A client of ours owns an IT company with 20+ locations across the UK. He is looking for a solution to provide each of their 20+ locations with a page or website that they can manage themselves that links directly back to the main site. His idea is to create 20+ one or two page websites that could all link back to the main central site - aiding the possibility of ranking well for locally-based terms. At the moment, we have a page for each of the 20+ locations on the main site. However, the client wants to give his franchisees complete control over their web presence. Would a setup like this work? Would it be logical to have 20+ websites (likely to follow a very similar format) all pointing to one central website? Would we have to "no-follow" links back to main site in order to show we aren't trying to manipulate page rank? Would creating sub folders on the main site be a better option for each of the 20+ locations? Any feedback appreciated!
Intermediate & Advanced SEO | | Webrevolve0 -
Best practice for removing indexed internal search pages from Google?
Hi Mozzers I know that it’s best practice to block Google from indexing internal search pages, but what’s best practice when “the damage is done”? I have a project where a substantial part of our visitors and income lands on an internal search page, because Google has indexed them (about 3 %). I would like to block Google from indexing the search pages via the meta noindex,follow tag because: Google Guidelines: “Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don't add much value for users coming from search engines.” http://support.google.com/webmasters/bin/answer.py?hl=en&answer=35769 Bad user experience The search pages are (probably) stealing rankings from our real landing pages Webmaster Notification: “Googlebot found an extremely high number of URLs on your site” with links to our internal search results I want to use the meta tag to keep the link juice flowing. Do you recommend using the robots.txt instead? If yes, why? Should we just go dark on the internal search pages, or how shall we proceed with blocking them? I’m looking forward to your answer! Edit: Google have currently indexed several million of our internal search pages.
Intermediate & Advanced SEO | | HrThomsen0 -
My site falls more than 50 in two days, help me!
My site appeared in the top 10 for that link (http://www.vipgoldrj.com/paginas/ensaios.html) and not by this (http://www.vipgoldrj.com), was well at 2 months, and he suddenly disappeared, I wanted to know if he had been penalized and Google told me it was not. What should I do? The. Sorry my English, I am Brazilian and I'm using Google translator. Warning from SEOmoz staff: this is an escort site with full frontal nudity and is not safe for most workplaces.
Intermediate & Advanced SEO | | WebMaster0210 -
How can I penalise my own site in an international search?
Perhaps penalise isn't the right word, but we have two ecommerce sites. One at .com and one at .com.au. For the com.au site we would like only that site to appear for our brand name search in google.com.au. For the .com site we would like only that site to appear for our brand name search in google.com. I've targeted each site in the respective country in Google Webmaster Tools and published the Australian and English address on the respective site. What I'm concerned about is people on Google.com.au searching our brand and clicking through to the .com site. Is there anything I can do to lower the ranking of my .com site in Google.com.au?
Intermediate & Advanced SEO | | Benj250 -
Need to duplicate the index for Google in a way that's correct
Usually duplicated content is a brief to fix. I find myself in a little predicament: I have a network of career oriented websites in several countries. the problem is that for each country we use a "master" site that aggregates all ads working as a portal. The smaller nisched sites have some of the same info as the "master" sites since it is relevant for that site. The "master" sites have naturally gained the index for the majority of these ads. So the main issue is how to maintain the ads on the master sites and still make the nische sites content become indexed in a way that doesn't break Google guide lines. I can of course fix this in various ways ranging from iframes(no index though) and bullet listing and small adjustments to the headers and titles on the content on the nisched sites, but it feels like I'm cheating if I'm going down that path. So the question is: Have someone else stumbled upon a similar problem? If so...? How did you fix it.
Intermediate & Advanced SEO | | Gustav-Northclick0