Robots.txt and redirected backlinks
-
Hey there,
since a client's global website has a very complex structure which lead to big duplicate content problems, we decided to disallow crawler access and instead allow access to only a few relevant subdirectories. While indexing has improved since this I was wondering if we might have cut off link juice. Since several backlinks point to the disallowed root directory and are from there redirected (301) to the allowed directory I was wondering if this could cause any problems?
Example: If there is a backlink pointing to example.com (disallowed in robots.txt) and is redirected from there to example.com/uk/en (allowed in robots.txt). Would this cut off the link juice?
Thanks a lot for your thoughts on this.
Regards,
Jochen
-
A noindexed page can still accumulate and pass link equity, although results vary on whether or not some of that link juice "evaporates" along the way. I'm inclined to agree with Chris, though, that there's probably no need to noindex a page that redirects to a page that you do want indexed.
-
Hi Jochen,
It's an interesting situation and to be honest, I don't know for sure how search engines will deal with that "link juice". This will come down to a question of whether search engines see robots.txt or htaccess first. If it looks at robots first (which is my suspicion), it can't see that page to pass the strength.
I suppose to test this, you could submit the redirected page to index via Search Console and see if it shows you the redirect or says it's blocked.
Interesting question aside, there's no real need to block access to a 301'd page
Also, apologies if I'm just highlighting the obvious here but it would be far better to clean up the site structure and remove that duplication rather than just masking it with robots; the user experience is at least as important as the algorithms!
Along the same lines, cleaning up those pages is going to help your crawl budget immensely.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ugly Redirect Chain
Hey everyone, Hoping to get your take on this: We have some very high demand products, they usually sell out in minutes (lucky us, eh?!) We are implementing a queue function on a product page - basically if too many people try to check out at the same time, we dump them in a queue The queue could kick in before or after search engines have indexed the product page The product page has markup and on-page content relating to the product. The queue page exists on an external (yes, external) site The queue page will not have any of the product info, markup, or optimised page title Product page will 302 to queue page and starts a series of 302 redirects! Here's the sequence when queue is active: CANONICAL product page (with markup, on-page product info, optimised page title, etc.)
Intermediate & Advanced SEO | | TSEOTEAM
>> 302 >> queue page on external domain (ZERO markup, product info or page title)
>>302>> same queue page, but throwing a hashed queue ID into the URL (basically giving you your place in the queue)
HELD IN QUEUE FOR A FEW MINUTES
**>> 302> ** NON-CANONICAL product page (with markup, on-page product info, optimised page title, etc.) I can foresee two scenarios search engine has indexed product page prior to queue kicking in. Then queue kicks in 302ing search engine to queue page. because it's a 302 the crappy queue page content is indexed back to the originating product page. This causes search engines to drop the product page cos all the product-specific markup/content has been overwritten with crappy queue page content search engines don't manage to index product page before queue kicks in. They crawl product page URL, get 302 to queue page, index crappy queue page content and think the product page is crappy, so don't traffic it. They will recrawl the product page once the queue's turned off, only to discover the product has sold out - boo. I very much doubt the search engines will 'wait for a few minutes' so may never end up reaching the product page again. I'm trying to get the markup/product info and optimised meta data injected into the queue page, so that remains present at all points on the journey in the hope that this enables search engines to continue to rank and traffic the product page. What's your take on this? Any suggestions on how we might overcome the issues? (before you ask; avoiding using the queue system is impossible, sorry!) Thanks!1 -
Backlink audit - anyone know of a good tool for manually checking backlinks?
Hi all, I'm looking at how to handle backlinks on a site, and am seeking a tool into which I can manually paste backlinks - is there a good backlink audit tool that offers this functionality? Please let me know! Thanks in advance, Luke
Intermediate & Advanced SEO | | McTaggart0 -
What does Disallow: /french-wines/?* actually do - robots.txt
Hello Mozzers - Just wondering what this robots.txt instruction means: Disallow: /french-wines/?* Does it stop Googlebot crawling and indexing URLs in that "French Wines" folder - specifically the URLs that include a question mark? Would it stop the crawling of deeper folders - e.g. /french-wines/rhone-region/ that include a question mark in their URL? I think this has been done to block URLs containing query strings. Thanks, Luke
Intermediate & Advanced SEO | | McTaggart0 -
Backlinks from old domain
Hi, We have gone through a change of company brand name including a new domain name.
Intermediate & Advanced SEO | | Agguk
We followed google recommendations at: https://support.google.com/webmasters/answer/83106?hl=en and it seems to have worked really well, the new domain has replaced the old in the google search results. My question: Still most of our backlinks, both anchor text and links use the old brand name and domain and it´s a slow process trying to update all references. Although they get redirected fine to the new domain (also following google recommendations), I wonder if the current scenario is doing any harm, SEO wise (other than the missed visual exposure of the new brand name) ? ...since the old brand name is not present at the new site I´m thinking of including "New brand name - previously old brand name" somewhere just to provide some sort of connection to all old backlinks, would that be unnecessary? I should mention that the old brand name actually includes our most important keyword but the new brand name does not. Thanks!0 -
Backlink Questions
Hey Mozzers, I have spent some time researching proper backlink analysis, and then I have been going through some of the steps. Here are a few questions that I have had in the process. Why would backlink tools like OSE and Ahrefs return different results for (say): "www.domain.com" vs "domain.com"? I noticed that competitors have almost 6x the backlinks as I do, but when I look at where those links are coming from, they are coming from old sites with moderate DA (under 10-30), but many are not current. I also noticed that many of these sites have links placed site-wide so that there are maybe 6+ referring pages per domain. So I guess my question is, how powerful are these links? Am I better off building relationships with bloggers, even though they only offer one link per page? Ultimately it will take me a long time to build the same quantity of links, but it seems like many of these competitors' links are old fashioned, but still moderately effective. Any help is appreciated, you guys have always been so helpful!
Intermediate & Advanced SEO | | evan890 -
301 redirect recommendations
One of our clients we are working on have two sites the main with a PR5 and a separate one with a PR4. We are planning on doing a 301 from the PR4 to a page on the PR5 Is it best to do: www.PR4.com ----> www.PR5.com/releveantPR4page or www.PR4.com/page ----> www.PR5.com/releveantPR4page Most pages on the PR4 site can fit into one PR5 page logically. However the PR4 has an about us, contact us, blog/with posts, FAQ, Applications, Legal Resources which are all pretty out dated.. The PR4 site is kinda messy and we are not sure if it will be easy to 301 each page individually with the user in mind. can we do a sitewide 301 redirect from the root PR4.com to a page PR/5.com/releveantPR4page and also do deeper 301's? PR4.com/PR4page ---> PR5.com/releveantPR4page
Intermediate & Advanced SEO | | Bryan_Loconto0 -
301 Redirects After Company Acquisition
We recently acquired a company, and now we are going to redirect all of the pages on their site to their respective pages on our site. Do we need to keep the original pages on their site active? For how long? Ideally, we would like to redirect everything and remove the old site entirely so we don't have to pay to keep hosting it. Is this possible? Thanks!
Intermediate & Advanced SEO | | pbhatt1 -
Setting up of 301 redirects
Good morning all, As part of the analysis of our website, we have realised that we are diluting our keyword strength in a particular area by having multiple zones all targeting the same keyword. We have decided to combine these zones into one, and set up 301 redirects so that the remaining zone gets the benefit of the other zones' link juice. When setting up a 301 redirect from zone "X" to zone "Y" say, do I need to keep all of the content in zone X, or should I remove all content before the redirect is set up? Does zone Y still get the benefit of zone X's link juice if the content is removed? Many thanks Guy
Intermediate & Advanced SEO | | Horizon0