Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Switching CMS and want to redirect links
Hi guys, I have a site that I run which has been around for a little while. It is actually in phpnuke, it also has a phpbb forum.. The site is so customized that I need to move it to something newer. So I am moving it to Joomla. I created a new domain just to get the joomla site created so I can move it to the main domain. My question is this. How should I go about switching from phpnuke to joomla without losing the links that are going to my site. Also, where can I find the links that are pointing to my site? Thanks for looking.
Link Building | | trumpfinc0 -
App SEO: do backlinks to your google play app help?
I have an app in the Google Play store and I am trying to improve it in the play store search rankings. I was wondering if back-links to our play store URL help? If so do today's rules around back-link quality matter or its it about volume? Any other advice would be welcomed. thanks
Link Building | | surfeasy0 -
Penguin,No warning message from Google
I'm losing traffic & I have been since April. I'm pretty sure it's Penguin but I have not received any messages from Google. Should I proceed with the disavowal tool to remove some of the spammy directory links? PS Submit Edge is owned by the Devil
Link Building | | KrisPhoto0 -
Is something strange going on with Google last few days?? Seeing strange results for competitive keywords
I am seeing some very strange results for competitive keywords. If I search on google.co.uk for "website design" which I feel is a pretty tough keyword. I see a site called http://www.techdesigns.co.uk/ If you go to the site its a laptop review site with no mention of website design on the site. They seem to be an Amazon Affiliate They have 23,597 links saying website design but mostly from adult websites. I have seen this on a few result related to SEO but dont want to mention any names in case we upset anyone on the forum. How can they rank for such a competitive keyword without onpage? Surely Panda should have hit this site? Why have they not been hit by Penguin so many adult links? Sorry if this is anyones site on here I am not looking to get you banned or hit just stumps me why this site ranks.
Link Building | | tempowebdesign0 -
Is it good practise to do a 301 redirect from domain.co.uk to www.domain.co.uk?
As the title says really, I'm sure the answer is Yes, but I cannot remember why probably one of the many things I read somewhere along the line and can't recall the reasoning can anyone enlighten me please?
Link Building | | RankStealer0 -
Following Penguin, what is the best practice for a web design agency placing a footer link in client sites?
For years we have included a 'Web Design by ...' link in the footer of all client websites designed by us. However, following the Penguin update, it seems that this legitimate practice could now be penalised. Should we be removing those links from our client's sites? There are tens of thousands, all with similar or identical text! For future client sites, should we avoid these footer links altogether? Or maybe limit the link to a single page on each site?
Link Building | | LewisJenkins691 -
Backllinks to our site but via redirect (still SEO helpful?)
Hi, A large website may link to us but via their internal redirect mechanism. So it would be somthing like this: [If you select the link, it eventually goes to our site but not from the original page. These links would be placed on lots of pages, but since they would all "pass through" the "redirect" page, it seems not as beneficial from an SEO perspective. Is this reasoning correct? Would google still follow the link and see where it ends up, even though it is via multiple pages? Ideally I'd prefer direct link but how much better is it than the redirect as listed above?](/redirect/rd?site=www.oursite.com/page&trackingparam=111111)
Link Building | | NicB10 -
Linking to Google Place page??
I'm wondering if anybody has tried and had success in improving search rankings by creating links to Google Place pages. What say you?
Link Building | | bdiddy0