Robots.txt and 301
-
Hi Mozzers,
Can you answer something for me please.
I have a client and they have 301 re-directed the homepage '/' to '/home.aspx'.
Therefore all or most of the linkjuice is being passed which is great. They have also marked the '/' as nofollow / noindex in the Robots.txt file so its not being crawled.
My question is if the '/' is being denied access to the robots is it still passing on the authority for the links that go into this page? It is a 301 and not 302 so it would work under normal circumstances but as the page is not being crawled do I need to change the Robots.txt to crawl the '/'?
Thanks
Bush
-
Thanks Yannick - really helpful
cheers and have a good weekend
Bush
-
Yes. Remove the noindex nofollow for / from the robots.txt. It doesn't add anything. It can only confuse SE's and lose you rankings.
-
Hi Yannick.
Thanks for your reply. The / redirect to /home redirect has been done and the / is excluded.
I'm limited as to what changes I can make. I can amend the robots.txt to crawl / which is 301's to /home and that way the links will be attributed and the 301 remains.
Will that be beneficial do you think?
Bush
-
I would suggest not excluding the / in the robots.txt.
First you redirect the / to home.aspx but then you tell SE's to exclude the home page which you are redirecting anyway. Makes no sense to me, so I would remove it in the robots.txt. No use anyway, because the url is being redirected.
I wouldn't use the 301 redirect to home.aspx. Why not make some (technical?) modifications so that the home page is available on the root of the domain? It's much more beneficial to you site's SEO. Or the other way around, redirect home.aspx to /. Because way more ppl will be linking to the root and not the home.aspx page.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Robots.txt allows wp-admin/admin-ajax.php
Hello, Mozzers!
Technical SEO | | AndyKubrin
I noticed something peculiar in the robots.txt used by one of my clients: Allow: /wp-admin/admin-ajax.php What would be the purpose of allowing a search engine to crawl this file?
Is it OK? Should I do something about it?
Everything else on /wp-admin/ is disallowed.
Thanks in advance for your help.
-AK:2 -
301 Redirects a Year Later
I inherited the digital maintenance of a website that was relaunched a year ago. In looking at Google Analytics, organic search a year later is still down 33%. I fear they did not install 301 Redirects but can't really get a specific answer from them. Is it possible to install them a year later to help with Google indexing and get back some of the organic traffic?
Technical SEO | | stansamples0 -
301 a whole subdomain to main domain
Hello, we have our website in two languages. The subdomain cs. for czech and main domain for english. Somehow this has caused google not to rank our main domain in local CZ SERP for many english keywords and we have not found out a reason other than it might think of them as duplicates or that we at the start had put targeting for the cs. domain to CZECH. (Its been 4 months, english keywords in Czech serp are easier and still it ranks them on the very last place, yet for example.com ranks them in google.com co.uk, fr etc in top 10) We will close our cs.example.com subdomain and use only example.com hoping this will not cause google to display only one domain and raise it in CZECH rankings. **Can i 301 redirect cs.example.com to example.com and all the categories of cs.example.com to relevant english categories. This should be ok? **
Technical SEO | | advertisingtech0 -
Robots.txt on http vs. https
We recently changed our domain from http to https. When a user enters any URL on http, there is an global 301 redirect to the same page on https. I cannot find instructions about what to do with robots.txt. Now that https is the canonical version, should I block the http-Version with robots.txt? Strangely, I cannot find a single ressource about this...
Technical SEO | | zeepartner0 -
301 and 302 for same link
Just trying to find out if this may be the root of a slight traffic dip and also if we should be redirecting differently. We relaunched and did 301 redirects to the new site, initially. Then, we decided to change from http to https. Our HTTP status now looks like this when using the MozBar: HTTP/1.1 301 Moved Permanently – http://site.com/oldurl
Technical SEO | | MichaelEka
HTTP/1.1 302 Found – https://site.com/oldurl
HTTP/1.1 200 OK - https://site.com/new Should we be changing that 302 to a 301? Are we losing link equity due to this? Thanks.0 -
A few misc Webmaster tools questions & Robots.txt etc
Hi I have a few general misc questions re Robots.tx & GWT: 1) In the Robots.txt file what do the below lines block, internal search ? Disallow: /?
Technical SEO | | Dan-Lawrence
Disallow: /*? 2) Also the sites feeds are blocked in robots.txt, why would you want to block a sites feeds ? **3) **What's the best way to deal with the below: - old removed page thats returning a 500 response code ? - a soft 404 for an old removed page that has no current replacement old removed pages returning a 404 The old pages didn't have any authority or inbound links hence is it best/ok to simply create a url removal request in GWT ? Cheers Dan0 -
How does robots.txt affect aliased domains?
Several of my sites are aliased (hosted in subdirectories off the root domain on a single hosting account, but visible at www.theSubDirectorySite.com) Not ideal, I know, but that's a different issue. I want to block bots from viewing those files that are accessible in subdirectories on the main hosting account, www.RootDomain.com/SubDirectorySite/, and force the bots to look at www.SubDirectorySite.com instead. I utilized the canonical meta tag to point bots away from the sub directory site, but I am wondering what will happen if I use robots.txt to block those files from within the root domain. Will the bots, specifically Google bot, still index the site at its own URL, www.AnotherSite.com even if I've blocked that directory with Disallow: /AnotherSite/ ? THANK YOU!!!
Technical SEO | | michaelj_me0 -
How do 301 redirects affect rankings?
Scenario: example.com/red-shoes gets 301 redirected to example.com/brown-boots because we have stopped selling red shoes and now only sell brown boots (which is a fairly new page with no authority). the red-shoes page ranked well for "red shoes" and "footwear". Will Google still index and show the red-shoes url in the SERPs? Will the "red shoes" and "footwear" keywords still rank well? Or does the redirected/new boots page need to properly support these keywords? The boots page has inherited the juice from the shoes page, but how does it help the boots page rank well? Only for keywords that both pages targeted, like a general "footwear" type keyword? Thanks in advance!
Technical SEO | | akim260