Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Subdomains, blogs and redirects, oh my!
Hello! We are currently hosting our blog on Medium which I would like to move away from. The options now are to host the blog with our custom CMS or use the HubSpot blog capabilities. I know their blog has built-in SEO best practices but I'm pretty sure it hosts the blog on a subdomain. I keep getting conflicting advice on this. In a whiteboard Friday Rand once said to NEVER host on a subdomain. But I know that google now associates links that redirect to the original site. I.e. If redpepperland.com redirects to repepper.land then redpepper.land should have all the link juice of redpepperland.com, right? But Moz doesn't recognize any inbound links to redpepper.land, I have to actually put redpepperland.com in as an additional site to see those links and the domain authority which worries me. So I guess two questions: 1. Does anyone have experience with Hubspot's blog? Will the subdomain hurt our SEO efforts? 2. Does google now know how to associate a redirect's link juice and domain authority to the page it redirects to? Not sure I know what I'm talking about at all but any help would be greatly appreciated!
Link Building | | redpepperland0 -
Are stackoverflow links follow or nofollow?
I've tried to find the answer to this question myself, but I've found differing opinions. The conclusion I've come to is that Stackoverflow allows follow links when the user that posts the link has sufficient reputation or if the link receives user validation. Has anyone else here used Stackoverflow.com that knows the answer to this question? 🙂
Link Building | | RG_SEO0 -
Can a No Follow Press Release harm you?
Hello I know that , in theory, a No Follow link should not be dangerous in any cause. My question is: Can a No Follow Press Release harm you? I watched the Matt Cutts Video about and was pretty clear that the answer is NO. I also read (Google Source) that all press releases should be no follow and with no rich anchor text. We agree till here, right? But what if the press release if part of a link network, bad quality or simply you have many many press releases from one of these PR networks? In this situation, the no follow link is still "safe" or do you think there is some danger? Im doing some "cleaning" and I would like to know your opinion about. Many Thanks for your help
Link Building | | AutoEurope0 -
301 redirect of PR 6 domain to PR 1 site, Will it improve my site's seo?
I have a pagerank 1 website but i recently purchased one old pagerank 6 domain, if I 301 redirect this pr6 domain to my currrent pr1 site, will it will increase my site's pr and improve seo ?
Link Building | | IndiaFPS0 -
Is anybody else noticing a dramatic change to their 'links to your site' section in Google Webmaster Tools?
Hey,
Link Building | | ChrisHolgate
Over the last six months or so we've been going through our backlink profile and cleaning up links from poor quality sources. Week by week there have been small changes in our Google Webmaster Tools 'links to your site' section to reflect this. I logged on this morning however and there has been a dramatic shift in the information displayed. Pretty much every bad link has been removed from the list including sites I know for a fact are still linking to us as they didn't communicate at all to our removal requests. Additionally, rather than showing the top 1000 links to our site as it used to, WMT is only showing 73 linking domains. The remaining 73 domains are good natural links from high quality sources. I'm guessing Google are just in the middle of an update and that the remaining linking domains (including the bad ones) will reappear shortly. This isn’t a request for advice or help but I’m just curious as to whether anybody else is seeing anything similar?0 -
How does Google treat multiple backlinks on the same page?
Hello mozCrew Just a quick yay or nay. We have an embeddable code on our site for customers who want to post our link on their website listed here: http://www.accupos.com/etc/embed-code-restaurants.php But in our link, there are actually 2 separate links, one branding link that says the name of our company, AccuPOS, and another link which targets a specific target keyword "Restaurant POS Software". My question is, does Google only give credit for the branding link, since it is listed FIRST? Or will we get double credit? Is this a good or bad practice, and what would you recommend? Thank you! Derek Moore
Link Building | | DerekM880 -
"gimme links" = easy to fill out profile/personal reputation links that offer a do-follow anchor text flexible link
I'm doing some SEO for small niche service sites...and aside from niche directories and some light article marketing and local search directories (yelp, etc) there really isn't a budget for content generation. I really only have a budget for profile/submission links. What are some sites where you can quite easily fill out a personal profile with an anchor text variable link that may not be THAT juicy but will stick (i.e. not a spam/fake profile type of link, but something like Linkedin). I'll start with giving away 1 www.istockphoto.com Any others that aren't obvious?
Link Building | | ilyaelbert0 -
Rankings on Google compared to Bing &Yahoo
Does anyone have any information about the differences between the search engines, im ranking #1 on google but only about 3rd or 4th on bing and yahoo. Cheers
Link Building | | eirianprice
EP0