How best to fix 301 redirect problems
-
Hi all
Wondering if anyone could help out with this one. Roger Bot crawler has just performed it's weekly error crawl on my site and I appear to have 18,613 temp redirect problems!! Rather, the same 1 problem 18,613 times.
My site is a magento store and the errors it is giving me is due to the wishlist feature on the site. For example, it is trying to crawl links such as index.php/wishlist/index/add/product/29416/form_key/DBDSNAJOfP2YGgfW (which would normally add the item to one's wishlist). However, because Roger isn't logged into the website it means that all these requests are being sent to the login url with the page title of Please Enable Cookies.
Would the best way to fix this be to enable wishlists for guests? I would rather not do that but cannot think of another way of fixing it.
Any other Magento people come across this issue?
Thanks, Carl
-
thanks. I completely forgot about the robots.txt method of fixing it. Will give it a go. I will add comparison in there too just to be safe.
-
Hey Carl
I have this problem a lot on my Magento stores since theres a ton of ways to access the same product and it can be an SEO nightmare. If you have FTP access I would recommend adding this to the robots.txt file:
Disallow: /wishlist/
This will stop the bots from crawling everything passed /wishlist/. Here's a good resource if you need further assistance: http://www.magentocommerce.com/knowledge-base/entry/setting-up-a-robotstxt-file
Hopefully that helps!
Good luck
-
Thanks will try and work out how to do that. Magento is a pain to make minor changes to.
Hopefully the crawl stats will look a little healthier next week !!
-
Use Rel="no index, no follow" on the link and roger will ignore the link. Job done
You wouldn't get any advantage of seeing the added items in someones wishlist as it would be duplicate content I'm presuming.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Redirects, 301's & 404's
I have tons of links that I have had added a redirect to after creating my companies new website. Is it bad to have all these 301s? How do I permanently redirect those links? Also, on Google Search Console it's telling me I have 1,000+ excluded links. Is this bad? Will it negatively affect me? Is this something to do with my sitemap? Any help would be greatly appreciated 🙂
Technical SEO | | sammecooper0 -
Switching forum software - 301 redirects?
Hi everyone I'm working on a successful Wordpress site that also has a forum attached. The forum currently uses YAF forum software, which requires Windows hosting. The site owner wants to switch to Linux hosting. This is not a problem for WP, but it does mean that we'll need to transfer the forum to Xenforo or something similar that runs on Linux. We're OK with the technical side of this, but we're worried about the SEO implications. The URL for every forum post (more than 50,000 of them) is going to change during this transfer. It seems completely impractical to 301 each of those, so should I just 301 the URLs that have inbound links? Also, what is google's algo going to think when we suddenly have ~50,000 404s? Many thanks in advance! J
Technical SEO | | van280 -
Existing content & 301 redirects
Hi All, I will try to keep this to the point. One of our websites was hit by penguin for unnatural linking. We are building a new site (same business, different domain), but we would like to take some of the pages/content off the old website and use it on our new site. Is it just a case of copying each page onto our new site and 301 redirect the old URL? Or should I just be completely rewording/recreating the old content so it is unique? Any help on this would be great, but I am also open to alternate methods too. Thanks Lewis
Technical SEO | | SO_UK0 -
Moving Blog and 301 Redirect Advice
Hello Moz Community, We recently moved our blog from its own domain to a directory on our website. We do not plan on moving over all the old blog posts because a majority most of them are based on events or time-sensitive information that has passed. We need advice on what to do with all of the old blog URL's? Should we just 301 all of them to the new blog directory on our website (www.domain.com/blog)? Should we take the time to move over all the old blog content and put the appropriate 301's in place? Any and all advice would be greatly appreciated. Thank you in advance.
Technical SEO | | All-Star-Vacation-Homes
Best,
Rich0 -
Hi, I am little bit confused in 301 redirect
Hi, I am little bit confused I have set my preferred domain to www but anyone can access my site via both www and non www domains, do I need to 301 redirect all non www to www or not , If yes then I want to know Why and If no then also I want to Why.
Technical SEO | | amarjitkapur0 -
How to know what pages are 301 redirecting to me?
Hi! It is easy to know if somebody is spam linking your website, looking i.e., looking at open site explorer to analyse the links profile. But, is it possible to know if a competitor of mine is redirecting a bad domain to main with a 301 redirect, thus transfering any bad SEO reputation to me? Best Regards, Daniel
Technical SEO | | te_c0 -
Permanent 301 redirects vs canonical urls?
Im moving a website that was .php to wordpress with a few static HTML pages. Which is better use permanent 301 redirects and delte the old pages, leave the old pages and use canonical urls and 301 redirects or something else?
Technical SEO | | senith0 -
301 redirects inside sitemaps
I am in the process of trying to get google to follow a large number of old links on site A to site B. Currently I have 301 redirects as well a cross domain canonical tags in place. My issue is that Google is not following the links from site A to site B since the links no longer exist in site A. I went ahead and added the old links from site A into site A's sitemap. Unfortunately Google is returning this message inside webmaster tools: When we tested a sample of URLs from your Sitemap, we found that some URLs redirect to other locations. We recommend that your Sitemap contain URLs that point to the final destination (the redirect target) instead of redirecting to another URL. However I do not understand how adding the redirected links from site B to the sitemap in site A will remove the old links. Obviously Google can see the 301 redirect and the canonical tag but this isn't defined in the sitemap as a direct correlation between site A and B. Am I missing something here?
Technical SEO | | jmsobe0