Robots.txt and 301
-
Hi Mozzers,
Can you answer something for me please.
I have a client and they have 301 re-directed the homepage '/' to '/home.aspx'.
Therefore all or most of the linkjuice is being passed which is great. They have also marked the '/' as nofollow / noindex in the Robots.txt file so its not being crawled.
My question is if the '/' is being denied access to the robots is it still passing on the authority for the links that go into this page? It is a 301 and not 302 so it would work under normal circumstances but as the page is not being crawled do I need to change the Robots.txt to crawl the '/'?
Thanks
Bush
-
Thanks Yannick - really helpful
cheers and have a good weekend
Bush
-
Yes. Remove the noindex nofollow for / from the robots.txt. It doesn't add anything. It can only confuse SE's and lose you rankings.
-
Hi Yannick.
Thanks for your reply. The / redirect to /home redirect has been done and the / is excluded.
I'm limited as to what changes I can make. I can amend the robots.txt to crawl / which is 301's to /home and that way the links will be attributed and the 301 remains.
Will that be beneficial do you think?
Bush
-
I would suggest not excluding the / in the robots.txt.
First you redirect the / to home.aspx but then you tell SE's to exclude the home page which you are redirecting anyway. Makes no sense to me, so I would remove it in the robots.txt. No use anyway, because the url is being redirected.
I wouldn't use the 301 redirect to home.aspx. Why not make some (technical?) modifications so that the home page is available on the root of the domain? It's much more beneficial to you site's SEO. Or the other way around, redirect home.aspx to /. Because way more ppl will be linking to the root and not the home.aspx page.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
To 301 or not to 301?
I have a client that is having a new site built. Their old site (WP) does not use the trailing / at the end of urls. The new site is using most of the same url names but IS using the /. For instance, the old site would be www.example.com/products and the new site, also WP, will be www.example.com/products/. WordPress will resolve either way, but my question is whether or not to go in and redirect each matching non / page to the new url that has the /. I don't want to leave any link juice on the table but if I can keep the juice without doing a few hundred 301s that certainly wouldn't suck. Any thoughts? Sleepless in KVegas
Technical SEO | | seorocket0 -
Google indexing despite robots.txt block
Hi This subdomain has about 4'000 URLs indexed in Google, although it's blocked via robots.txt: https://www.google.com/search?safe=off&q=site%3Awww1.swisscom.ch&oq=site%3Awww1.swisscom.ch This has been the case for almost a year now, and it does not look like Google tends to respect the blocking in http://www1.swisscom.ch/robots.txt Any clues why this is or what I could do to resolve it? Thanks!
Technical SEO | | zeepartner0 -
How to solve the meta : A description for this result is not available because this site's robots.txt. ?
Hi, I have many URL for commercialization that redirects 301 to an actual page of my companies' site. My URL provider say that the load for those request by bots are too much, they put robots text on the redirection server ! Strange or not? Now I have a this META description on all my URL captains that redirect 301 : A description for this result is not available because this site's robots.txt. If you have the perfect solutions could you share it with me ? Thank You.
Technical SEO | | Vale70 -
How do find where a 301 redirect is located
My report says I have http://www.30minuteseder.com/Passover.blog redirected to http://30minuteseder.com/Passover.blog. It is correct, but I can't find where the 301 redirect is located. I looked in my .htaccess file in the root and it's not there. How do I find it so I can change it?
Technical SEO | | Sederman0 -
Removing robots.txt on WordPress site problem
Hi..am a little confused since I ticked the box in WordPress to allow search engines to now crawl my site (previously asked for them not to) but Google webmaster tools is telling me I still have robots.txt blocking them so am unable to submit the sitemap. Checked source code and the robots instruction has gone so a little lost. Any ideas please?
Technical SEO | | Wallander0 -
When is it safe to remove 301 redirects?
I have created over 500 301 redirects in my .htaccess file, some of them are more than 2 years old now. Should I delete them? I don't like seeing the "notices" number in crawl diagnostics so high 😞
Technical SEO | | danielshaw0 -
301 redirect on wordpress.com
Hi, How/where do I input a 301 redirect for my http site within wordpress.com. I am already redirecting my wordpress url to myy new non wordpress domain Thanks Lisa
Technical SEO | | lisamarieCunn0 -
How should 301 redirects affect Page Authority?
We recently setting up 301 redirects from one of our sites so that the site redirects from the www version to the non-www version for all pages. We want to quantify what we expect to see as results. From what the experts say, we'd expect that the Page Authority of the canonical versio (non-www) will be higher than either of the two separate ones were previously. For instance, if this page - www.website.com/information/ - had a PA of 57 and this one - website.com/information/ - had a PA of 53, some time after the 301 redirects from www to non-www have been put into place, we should see the non-www version of that page move up to some PA about 57. It our thinking correct? How long does it normally take to see a PA update take place in a scenario like this? Thanks, Richard
Technical SEO | | LDS-SEO0