Why is Google not following a 301 redirect on the robots.txt file?
-
Hi Guys,
I recently posted a question on the Google Webmasters Forum http://www.google.com/support/forum/p/Webmasters/thread?tid=683e71557db7fd54&hl=en&fid=683e71557db7fd540004a4b4add8cbb6 and didn't get a satisfactory feedback so I thought I will put this to the SEO gurus on here. Perhaps one of you guys might good buddies with Matt and might be able to ask him directly. I actually posted on Matt's blog but he hasn't got back to me.
Basically we did a URL restructure for client and set up 301 redirects and saw a huge drop in rankings over time. The 301 redirects seem to work fine and have been tested by many many people. We suspected that google might be ignoring the 301 redirects or devaluing them and so I reviewed the server log to see what is happening when the googlebot crawls the site and it showed that on many occasions the googlebot did not reload the page after hitting a 301 redirect.
Sure.. you might say it probably queues it or Google might just be checking that the 301 redirect is still in place but why check so often (with a few hours to a day on the same URL) it even skips a 301 redirect on the robots.txt file i.e. from http://clientsite.com/robots.txt to http://www.clientsite.com/robots.txt. from non-www to www version.
I don't think it is easy to dismiss the skipping of the robots.txt file - this 301 redirect should be loaded immediately to use the instructions the gooblebot requires to crawl the page.
Any help will be appreciated. I can sent the server log to anyone personally but I am reluctant to post it on here.
Regards,
Zan
-
Thanks John, I check the robots.txt redirect and it is redirecting fine from the non-www version to the www version. I also know that G is crawling the site because we can see requests by G in the server log. Also as with the case with the other 301 redirects there were cases G followed the 301 redirect on the robots.txt file but also many cases where it didnt.
-
Hi Zan-
Check out this thread:
http://forums.searchenginewatch.com/showthread.php?t=23404
Reading it through, I thought comments #8 and #11 might be of interest to you.
-John
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
slug Link redirect to subdomain?
Hi !
Link Building | | Leviiii
Im Levi new here and new in the world of SEO, please dont judge if my questions are silly. Back on the days when the site was built we thought it is a good ideea to have subdomains that together with the domain name represent our main keywords.
ex. https://stansted.tonorwich.uk, https://heathrow.tonorwich.uk, https://luton.tonorwich.uk, https://gatwick.tonorwich.uk. There is content on this subdomains, would it make any difference from SEO perspective if we create slugs that redirect to these subdomains? for example creating https://tonorwich.uk/taxi-minibus-vip-tesla-norwich-to-stansted that redirects to https://stansted.tonorwich.uk ? Or better create these slugs with slightly different content?
Any ideeas would be appreciated.
Thanks in advance!0 -
Do i have too many 'follow' backlinks and am i being penalised by Google for it?
Hi all. I read on Moz recently that if a website has too large a percentage of 'follow' backlinks, that Google penalise the website because that is unnatural. IS this correct please? I ask because i have recently found that our own website, according to Moz, has 16,500 inbound links and they are ALL 'follow' links. These are all from independent 3rd parties and we havent commissioned any of them, so it is completely natural. URL if anyone cares is www.themosquito.co.uk Any advice would be appreciated. Cheers
Link Building | | TheMozzy0 -
Connecting Two Websites Without Upsetting Google
I have a question that I think I know the answer to, but want to cover my bases. We have a successful blog and are creating a second similar blog for a more specific type of reader. We want to backlink between content because it makes sense for the reader. Our fear is that we may come off spammy to Google by having so many backlinks. Is there a way we are able to tell Google these two sites are related so they don't penalize either site for anything?
Link Building | | HashtagJeff0 -
Google Penalty due to external links
On Sep 3, we got a Manual Action by Google: unnatural links to your site Immediately we arranged to have all those links the rel="nofollow" atttributes, these were links set by our customers since we are hosting providers. We sent a reconsideration request to Google on Sep 8 and this is their answer:
Link Building | | netbuilder
We’ve reviewed the reconsideration requestfor xxxx and modified its status, but we still believe that content on your site or links to your site are outside our quality guidelines. No more info provided. - We double checked everything but could not find anything else that could harm us so we asked Google (in a new reconsideration request sent on Sep 11 to provide us some samples. On Sep 24 Google answers with a sample of 3 sites which contains links to us. After verification, all the external links to us have the rel="nofollow" attribute !! Today the manual action is not anymore a side-wide one but a partial one and applies to unnatural links--impact links. So it seems that we need to physically have those links removed? As well I checked https://support.google.com/webmasters/answer/2604772?ctx=MAC and he does not refer at all about setting the rel="nofollow" attribute, Looks like Google wants the full monopoly on text links.0 -
Google Panda Update question
Since the Panda update from December 21st, 2012, some of the keywords I used to rank for are now gone beyond page 20. What I don't get is that the websites that are now ranking are definitely full of crap and they have tons of unnatural links pointing to them. I've analysed their backlink profiles and most of them have only exact keyword match anchors pointing to them. So what's going on? Is buying tons of crappy links still a valid tactic? I know it is not but how come thoses sites outrank me? I mean it happened to LOTS of my keywords. I don't get it...
Link Building | | sbrault740 -
Using brand domain for easy verbally said with 301 redirect to exact match.net for a site
I have recently bought an exact match.net for a keyword with no hyphens (on the fence about hyphens, read so many different views on here). From looking around it seems that not all solicited links that I am interested in use anchor text, but more significantly exact match domain would be like automatic anchor text when anyone from the general population posts a link to your site. I sell print books and am thinking of keeping the non-keyword domain in them so they are easy to say and type, while using the exact match domain for everything else online which the non-keyword domain would 301 redirect to. I don't do any programming and will use cPanel on my webhost to do the 301 redirect, I already have one 301 to go to the www subdomain so that would make two 301's. This might sound quite straight forward to many here but it's fairly new to me and I've not seen cPanel mentioned here yet, so was wondering if I am going about it the right way.
Link Building | | Zoolander0 -
No internal followed links
My site shows no internal followed links in OSE, but each page on my site has internal links, including to the domain itself - jonnyt.me. Any idea why this might be and how I could fix it?
Link Building | | jontarbuck0 -
Retain keywords with redirect
I manage a site the sells product A and product B (both major keywords for us). If I buy the url www.productB.com and redirect it to the original website, would the original website gain any benefits in search engines? Would the benefits of having a url with the keyword be lessened because there isn't content hosted there, only a redirect to the original site?
Link Building | | livethesource0