Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Ranking
I'm working on a page for couple of months now and try to rank for very competitive keyword. I have reached position 12 and it's 97% optimized, the sites keeps going up and down between position 10 & 13. I already changed content a couple of times and already created a few links with the targeted anchor tags. However I'm scared if I create more links it will be penalized. What do you suggest? Should I create more links with the targeted anchor tags or should combine the keyword in anchor tags with other words? Loading time, title, descriptions, content everything perfect optimized.
Link Building | | bill3690 -
Unnatural link Manual Penalty by Google
Hi, Our product Quiz Maker help users to create quizzes and can share, embed etc. We have a link in our embed code which was DoFollow and we've got many doFollow links from the websites on which our quizzes got shared but these links are too old like 2009, 2012 etc. After that, to complying with Google policies, we've changed the dofollow link to nofollow in embed code. Here are two scenarios: 1.) We've too many dofollow links which are too old and we don't have control over it as these are the natural links, given by our users before we changed the embed code link. 2.) Now, we are getting nofollow links from the widget as the embed code consists of nofollow link. Our embed code is editable and user can change it to dofollow or nofollow as per their requirement. Moreover, some people shared quizzes on their website (without using embed code) with dofollow link. But, now, google is taking manual action penalty against us. How we can resolve it as it seriously affects the overall performance of our website. Thanks
Link Building | | SameerBhatia2 -
Do-follow links from social media
I am experimenting with some blogs and online communities for backlinks. The Moz bar shows this link as do-follow, but it has the internal redirecting link. My brain is frozen and I can't remember what it's called to look up how it affects SEO. Can anyone enlighten me?
Link Building | | julie-getonthemap0 -
Citation, Google Plus and SEO questions
Hi, I have a website www.gemslearninginstitute.com. We offer different after school programs for kids located in Canada. I have a very basic knowledge of SEO. I have a few questions. Just wondering if someone can please help me. We offer UCMAS Mental Math Program, Tutoring and Chess School/Coaching and runs a Chess Club. I would like to boost my google ranking and get more local traffic. 1- My citation is not consistent. Which program should I use to correct the citation and delete duplicate or inconsistent listing automatically. 2 - Should I do the citation with my company name only “GEMS Learning Institute” or Should I do like this “GEMS learning Institute – UCMAS-Tutoring-Chess”. Which one is better? 3- Right now I have two different google plus business pages. One with the name of “GEMS Learning Institute” and the second one is UCMAS. Should I have only one page as GEMS and delete the UCMAS. 4- I am not sure how to check if my site is penalized by google. 5- How can I measure things which are working on not? Can you please recommend next steps should I take after doing citation to improve the ranking? Thanks in advance and appreciate help.
Link Building | | fslpso0 -
Links from PRWeb press release violate Google's quality guidelines?
My site has had a manual action performed on it by Google indicating that I have inbound links that fall outside of their quality guidelines. I did my own research, found what I thought was the issue, had the links removed and requested reconsideration. Google's response surprised me in that they highlighted two specific pages with links that were the direct result of valid press releases and a publisher picking up our release off a wire service. Has anyone else seen this occur? Anyone had a case successfully reconsidered? I realize that I don't need to do anything at all as the manual action is in effect and will stay that way, discounting those links, but I would rather a) not have any manual action against my site and b) know for the future so this doesn't happen again. Also, is this applicable for guest blog posts, which effectively create the same type of backlinks? Thanks
Link Building | | barberm1 -
Google Display Network
Hello! Ive been analyzing my competitors backlinks and see a large number of their google display ads shows as (alt text). I do have a google adwords account but stayed away from it in the past opting to use the search network only to keep cost down. Do the images count as a backlink if I can see them in the Seomoz tool? Thank You!!
Link Building | | TP_Marketing0 -
Last week the Google Australia ranking for one of my keyword was 10, this week it slipped to 20\. How come ?
Search Engine: Google Australia (organic) Keyword: Staging Hire Previously Ranked: 20 After several article and link submissions it jumped to 10. But in just one week slipped back to 20. Same thing happened with few other keywords too. Why?
Link Building | | i-kreo0 -
How often is a page indexed by google.
Are there any rules according to PA/DA how often a page is visited by google and how quick links there are picked up? For instance PA42/DA37 mR and mT both around 4,4 Thanks
Link Building | | danlae0