Longevity of robot.txt files on Google rankings
-
This may be a difficult question to answer without a ton more information, but I'm curious if there's any general thought that could shed some light on the following scenario I've recently heard about and wish to be able to offer some sound advice:
An extremely reputable non-profit site with excellent ranking had gone through a re-design and change-over into WordPress. A robots.txt file was used during development on the dev site on the dev server.
Two months later it was noticed through GA that traffic was way down to the site. It was then discovered that the robot.txt file hadn't been removed and the new site (same content, same nav) went live with it in place. It was removed and a site index forced. How long might it take for the site to re-appear and regain past standing in the SERPs if rankings have been damaged. What would the expected recovery time be?
-
They were paying attention to GA but lapsed and when they checked back in, saw a drop in traffic. Great point about that "critical" message.. The developers did force a crawl and I'm hoping you are correct about the time it might take.
-
Thank you methodicalweb. Great suggestions.
-
Thanks, Travis. You've offered a lot of very interesting points.
I will double-check that they have looked at the server log files, but I'm pretty confident that they have done that.
They did assure me that the proper redirects were done but I'm not sure what they did regarding extensions. There was also a server change.....
-
Thanks for clarifying KeriMorgret. Much appreciated. As are all your thoughts. I will definitely suggest that the monitoring software be used to avoid any future problems. This was such an unnecessary and frustrating experience.
-
If they were paying attention to WMT they would have seen a "critical" message that the site was blocked right away. Forcing a crawl (crawl all urls) should result in the site getting indexed extremely quickly. Rankings should return to where they were before.
-
The only thing I would add to the existing responses, is that if following a "site:www.mysite.com" query you notice that some key landing pages haven't been indexed then submit them via Webmaster Tools (Fetch as Google).
I would also make sure your sitemap is up to date and submitted via WMT too. It will also tell you how many of the sitemap URLs have been indexed.
These 2 things could speed up your re-indexing. My guess is that if it's a reputable site, and the migration of URLs was done properly, you'll probably get re-indexed quickly anyway.
George
-
Hi Gina,
Yes, that is what I mean. The dev team (or you, if you chose) would get an email that says the robots.txt file had changed. I was inhouse at a non-profit where we had an overseas dev team that wasn't too savvy about SEO, so I was the one who would get the emails, then go and send them an email asking them to fix it.
I don't believe there's a hard and fast answer here, as it in part depends on how quickly your site is crawled.
-
If possible, take a look at the server log files. That should give you a better idea of when/how often Google crawled the site in recent history. The user agent you're looking for is googlebot.
Aside from the robots.txt faux pas, it's also possible that the proper redirects weren't put in place. That would also account for a dip in traffic. Generally WordPress is extensionless. Which means any previous URL that contained an extension won't properly resolve - which means the site would lose a chunk of referral traffic and link equity if the URLs contained an extension (.php, .html, .aspx). Further, if the URL names have been changed from something like /our-non-profit.html to /about-our-non-profit those would require a redirect as well.
I've seen brand new domains index in a matter of days, then rank very well in as little as one month. But that's the exception, not the rule.
Provided proper redirects are in place and nothing too drastic happened to on-page considerations, I would guesstimate two weeks to a month. If you start heading into the month time frame, it's time to look a little deeper.
edit: If the server changed, that would also add another wrinkle to the problem. In the past, one of my lovely hosts decided to force a change on me. It took about a month to recover.
-
Thanks so much for your response KeriMorgret. I'm not sure I fully understand your suggestion unless you are saying that it would have alerted the dev team to the problem? I will pass this on to them and thank you if that is what your intention was.
The developer removed the robot.txt file which fixed the problem and I am trying to ascertain if there is a general expectation on how something like this - a de-indexing - gets reversed within the Google algorithm.
-
I don't know how long it will take for reindexing, but I do have a suggestion (have been in a real similar situation at a non-profit in the past).
Use a monitoring software like https://polepositionweb.com/roi/codemonitor/index.php that will check your robots.txt file daily on your live and any dev servers and email you if there is a change. Also, suggest that the live server's robots.txt file be made read-only, so it's harder to overwrite when updating the site.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to activate google optimize free version?
HI, How to activate https://www.google.com/analytics/optimize/ free for my ecommerce site? I am already using google analytic and google tag manager? So this is best tool for doing A/B testing? It is free for 30 days only? Thanks
Reporting & Analytics | | dsouzac0 -
Buffer Link and Google Impressions
Afternoon, I noticed a spike in impressions over a couple of days in April, so I investigated Analytics to see where these were coming from. It appears these impressions were split between two URLs; one was a blog post, the other was the Buffer link to the blog post that we used on Twitter and Facebook. According to Analytics, this Buffer URL received 1000 impressions over two days, with an average SERP position of 16. This surely can't be right, can it? Is this just another Analytics quirk? After two days of a decent amount of impressions to this Buffer link, the amount of impressions dropped to pretty much zero. I know Tweets are now starting to rank, but this would be the Twitter URL, not the Buffer link to our blog post? Any ideas, Cheers, Lewis
Reporting & Analytics | | PeaSoupDigital0 -
Is Google still differentiating TLD?
I need to find an article or something with factual findings that shows Google no longer gives extra value to .com, .net, .us, edu. etc. or proving that is still does.
Reporting & Analytics | | PPI0 -
Google Analytics Organic Accuracy?
Hi, On a cool summers day, I was quenching my thirst for SEO Knowledge in the depths of Google analytics, When I stumbled upon a most troublesome sight. I was Browsing the data inside Acquisition->Keywords->Organic and comparing them to Acquisition->Search Engine Optimization->Landing Pages, to find the results quite questionable.. Since Keywords->Organic measure the visits from the organic results, and Search Engine Optimization->Landing Pages measures the clicks onto one of my landing pages from Google, I assumed they should be the same or very close, (besides Keywords->Organic having visits from other search engines) but they were not the same, as a matter of fact it was telling me that Keywords->Organic visits = 1,474, 27.19% of total(5,422) and Search Engine Optimization->Landing Pages, clicks = 1,548 154.80% of total (1000). Even worst the Search Engine Optimization -> Queries clicks = 1,123, 160.43% of total(700). I have been searching, and come up empty handed in the answer so does anyone know why Google is showing me different results and strange numbers that don't make sense? Any information is greatly appreciated?
Reporting & Analytics | | KBB_Digital0 -
Are RSS fees tracked by Google Analytics?
In both the new GA and former version, how are RSS feeds handled by Google Analytics. If not, is there an easy means by which to account for this traffic, in order to have a better picture of traffic. Thanks. Alan
Reporting & Analytics | | ahw0 -
Google analytics vs Webmaster tools data
Hi Which is more accurate WT or GA data ? Since GA reporting a KW (thats very recently fallen from page 1 to 3 hence looking into data to find the cause) in the Organic part of Search tab as having generated just 1 visitor over a month (hence presuming fall could be due to low visits from a page 1 result) whilst under Search Engine Optimisation tab (data sourced from WT i think) its reporting 5 click thrus from 150 impressions over same period resulting in a quite good 3.33% CTR (hence wouldn't expect to be the cause of a fall) and what i would have thought GA would report as 5 visits instead of the 1 they do report !? The reason im looking for answer in the data is because no on-page has changed and still scoring a grade A and off page metrics have all improved across the board (apart from small drop in majseo's Trust Flow) such as increased links, RD, Citation Flow, Ref Subnets etc etc etc Cheers Dan
Reporting & Analytics | | Dan-Lawrence0 -
Google Analytics and DNS change
Our new alumni application is going be tested at domain uva.imodules.com . We are going to collect traffic data with a Google analytics account number UA-884652-XX. So going to uva.imodules.com/myPage.html would send its data to Google Analytics with that account number. Then when it is ready for production we are going to just change the domain name of the application and switch the DNS over to dardencommunity.darden.virginia.edu . So going to dardencommunity.darden.virginia.edu /myPage.html would send its data to Google Analtics with that SAME account number. Aside from having the testing domain data in the same profile are there any other issues/problems we may run into?
Reporting & Analytics | | Darden0 -
Tool to check pervious rankings for a keyword ?
I am looking a tool/site to check ranking history of specifc keyword . Is this possible ? Thanks
Reporting & Analytics | | bfletc40