Solve duplicate content issues by using robots.txt
-
Hi,
I have a primary website and beside that I also have some secondary websites with have same contents with primary website. This lead to duplicate content errors. Because of having many URL duplicate contents, so I want to use the robots.txt file to prevent google index the secondary websites to fix the duplicate content issue. Is it ok?
Thank for any help!
-
yes, robots.txt is the bad way, I will try to use canonical tag. Thanks for your help!
-
Using robots.txt is perhaps not the best way of doing it. Using the canonical or a noindex meta tag would likely be best. I think the reasons for this are best summed up in this article which explains, probably better than I could, why robots.txt is not the best way of dealing with duplicate content. Hope this helps.
-
I have tried to use cross-domain canonical, but this is too difficult for me. So, I want to confirm if I use the robots.txt file is ok or not?
Thanks
-
Why not use a cross-domain canonical whereby you reference the pages on your primary website as the canonical version on your secondary websites, thereby eliminating the duplication.
For example on each page that is duplicate on your secondary website you would add the following to the head to reference the primary pages:
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What constitutes duplicate content on a page?
I am working on SEO for a Shopify store. Their products are very similar, hence the pages are so similar that Moz shows them as duplicate content. The only difference in the product pages is the title and model number. I am going to "go for the gold" and try re-writing all the product descriptions. It's incredibly difficult due to the products being nearly identical with just a minor variation. I know I could go down the road of just creating variants --- but the customer is not down for that. Here's my question: what constitutes duplicate content? 80% of the content, 90%???? If I can going to re-write the descriptions, what should I aim for? Thank you!
On-Page Optimization | | steve_linn1 -
Duplicate content affects on overall rankings
Hi guys, I have a website that has 23 pages with duplicate content. These pages serve the same function, which enables customers to upload their images. There is not much content on each one but we require a different page for each of our products, here is an example page: http://www.point101.com/giclee_printing/upload#/upload I don't think it makes sense to use a canonical tag as each page is for a different product and I think its going to be difficult to differentiate each page. I was wondering: 1. If this has a negative effect on the ranking of our homepage and other main product pages or if its an issue we do not need to worry too much about. 2. If anyone has any other ideas as to how we can resolve this issue. Thanks,
On-Page Optimization | | KerryK
Kerry0 -
Which permalink structure should i use
Hi Im using wordpress for a trading market update site so most of the posts i add are time relevant... What permalink structure is best to use? Just /postname/ or /year/month/postname/ Thanks in advance! Chris
On-Page Optimization | | chrisleonard0 -
Is this duplicate content okay?
We have a client who wants to rank locally, nationally and internationally for their products. I wrote a line that goes, "We can ship our products to you whether you’re here in Illinois, nationwide, or international." I added that line after a paragraph or two of unique product description on each of their 30-odd product pages. Will this damage their ranking? I tried researching this but only found full page duplicate content topics. Any advice would be great.
On-Page Optimization | | optimalwebinc0 -
Mobile blog creating duplicate title and content issues
My site is set up with a mobile website and after running the on page SEO report on moz it is returning multiple errors for duplicate title tags and duplicate content all related to the mobile website being a direct reflection of the main site. I want to know if this is really an issue. I'm sure Google actually knows the difference between my main site and the mobile version. Do I really need to go through each individual mobile page and change each page title and modify all the content? I really would like to avoid doing this but at the same time would like to fully optimize my site. Any advice is appreciated.
On-Page Optimization | | shiftins0 -
Meta Data definition for multiple pages. Potential duplicate content risk?
Hi all, One of our clients needs to redefine their meta title and description tags. They publish very similar information almost every day, so the structure they propose is the following: Structure 1: Type of Analysis + periodicity + data + brand name Examples 1: Monthly Market Analysis, 1/5/2012 - Brand Name Weekly Technical Analysis, 7/5/2012 - Brand Name Structure 2: Company Name + investment recommendation + periodicity Example 2: Iberdrola + investment recommendation (this text doesn't vary) + 2T12 (wich means 2012, 2nd trimestrer) Regarding meta description they want to follow a similar approach, replicating every time the same info with a slight variation for each publication. I'm afraid this may cause a duplicate content problem because of the resemblance of every "Market Analysis" done or every "Investment recommendation" done in the future. My initial suggestion for them is to define specific and unique meta data for each page, but this is not possible for them given the time it takes to do it for every page. Finally, I ask them to specify the data in each meta title of content published, in order to add something different each time and avoid duplicate content penalty. Will this be enough to avoid duplicate content issues? Thanks in advance for your help folks! Alex
On-Page Optimization | | elisainteractive0 -
How much constitutes duplicate content in your opinion?
Mornin' In your experience, how much constitutes duplicate content? A sentence, a paragraph, half a page, etc? What about quotes - are they considered duplications, too, if there aren't quotation marks? Over the years, the client has been a bit bad in taking a paragraph from here, a sentence from there, and coupling it all together as daily news on their site. I'm now in the middle of a purge. Oh boy! All hail originality.
On-Page Optimization | | Martin_S0 -
Duplicate content issues with products page 1,2,3 and so on
Hi, we have this products page, for example of a landing page:
On-Page Optimization | | Essentia
http://www.redwrappings.com.au/australian-made/gift-ideas and then we have the link to page 2,3,4 and so on:
http://www.redwrappings.com.au/products.php?c=australian-made&p=2
http://www.redwrappings.com.au/products.php?c=australian-made&p=3 In SEOmoz, they are recognized as duplicate page contents.
What would be the best way to solve this problem? One easy way i can think of is to nominate the first landing page to be the 'master' page (http://www.redwrappings.com.au/australian-made/gift-ideas), and add canonical meta links on page 2,3 and so on. Any other suggestions? Thanks 🙂0