Severe rank drop due to overwritten robots.txt
-
Hi,
Last week we made a change to drupal core for an update to our website. We accidentally overwrote our good robots.txt that blocked hundreds of pages with the default drupal robots.txt. Several hours after that happened (and we didn't catch the mistake) our rankings dropped from mostly first, second place in Google organic to bottom and mid first page.
Basically I believe we flooded the index with very low quality pages at once and threw a red flag and we got de-ranked.
We have since fixed the robots.txt and have been re-crawled but have not seen a return in rank.
Would this be a safe assumption of what happened? I haven't seen any other sites getting hit in the retail vertical yet in regards to any Panda 2.3 type of update.
Will we see a return in our results anytime soon?
Thanks,
Justin
-
Your present approach is correct. Ensure all these pages are tagged as noindex for now. Remove the block from robots.txt and let Google and Bing crawl these pages.
I would suggest waiting until you are confident all the pages were removed from Google's index, then check Yahoo and Bing. If you decide that robots.txt is the best decision for your company, then you can replace the disallows after confirming your site is no longer affected by these pages.
I would also suggest that, going forward, you ensure any new pages on your site that you do not wish to index always include the appropriate meta tag. If this issue happens again then you will have a layer of protection in place.
-
We're pretty confident thus far that we have flooded the index with about 15,000 low rank URLs all at once. This has happened once in the past a few years back but we didn't flood their index, they were newer pages at the time in which were low quality and could have been seen as spam since there was no real content but adsense so we removed them with a disallow in robots.
We are adding the meta no-index to all of these pages. You're saying we should remove the disallow in robots.txt so googlebot can crawl these pages and see the meta-noindex?
We are a very large site and we're crawled often. We're a PR7 site and MOZrank DA is 79/100. We have dropped from 82.
We're hoping these URLs will be removed quickly, I don't think there is a way of removing 15k links in GWMT without setting off flags also.
-
There is no easy answer for how long it will take.
If your theory about the ranking drop being caused by these pages being added is correct, then as these pages are removed from Google's index, your site should improve. The timeline depends on the size of your site, your site's DA, the PA and links for these particular pages, etc.
If it was my site I would mark the calendar for August 1st to review the issue. I would check all the pages which were mistakenly indexed to be certain they were removed. After, I would check the rankings.
-
Hi Ryan,
Thanks for your response. Actually you are correct. We have found some of the pages that should be no follows still indexed. We are now going to use the noindex, follow meta tags on these pages because we can't afford to have theses pages indexed as they are particularly for clients/users only and are very low quality and have been flagged before.
Now, how long until we see our rank move back? Thats the real big question.
Thanks so much for your help.
Justin
-
That's a great answer Ryan... I wonder, just out of curiosity, if it wouldn't hurt to look at the cached version of the pages if they're indexed? I'd be curious to know if the date they were cached is right near when the robots.txt was changed? I know it wouldn't alter his course of action, but might add further confirmation that this caused the problem?
-
Justin,
Based on the information you provided it's not possible to determine if the robots.txt file was part of the issue. You need to investigate the matter further. Using Google enter a query in an attempt to find some of the previously blocked content. For example, let's assume your site is about SEO but you shared a blog article about your movie review of the latest Harry Potter movie. You may have used robots.txt to block that article because it is unrelated to your site's focus. Perform a search for "Harry Potter insite:mysite.com" replacing mysite.com with your main web address. If the search returns your article, then you know the content was indexed. Try this approach for several of your previously blocked areas of the website.
If you find this content in SERPs, then you need to have it removed. The best thing to do is add the "noindex, follow" tags to all these pages, then remove the block from your robots.txt file.
The problem is that with the block in place on your robots.txt file, Google cannot see the new meta tag and does not know to remove the content from it's index.
One last item to mention. Google does have a URL removal tool but that would not be appropriate in this instance. That tool is designed to remove a page which causes direct damage by being in the index. Trade secrets or other confidential information can be removed with this tool.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Lost ranking after domain switch
I recently migrated from https://whitefusemedia.com to https://whitefuse.com. The website URL structure and content remained the same and I followed all the best practice guidance regarding checks on the new domain and appropriate 301 redirects. I have seen traffic drop by about 50% and the traffic that is still coming through is mainly coming through links still listed by Google under the old domain (https://whitefusemedia.com). Is this normal? Should I expect to see this bounce back, or is there anything I can do now to regain the rankings?
Technical SEO | | wfm-uk0 -
Is sitemap required on my robots.txt?
Hi, I know that linking your sitemap from your robots.txt file is a good practice. Ok, but... may I just send my sitemap to search console and forget about adding ti to my robots.txt? That's my situation: 1 multilang platform which means... ... 2 set of pages. One for each lang, of course But my CMS (magento) only allows me to have 1 robots.txt file So, again: may I have a robots.txt file woth no sitemap AND not suffering any potential SEO loss? Thanks in advance, Juan Vicente Mañanas Abad
Technical SEO | | Webicultors0 -
Authority site drastic ranking drop after google https switch. Please Help!
Hi Mozers. Since Google switched to the https version, our company website (http://we.register.it) indexing switched to the https version (https://www.register.it). After that the ranking on Google dropped for almost every keyword. The site is very old and got a great authority and PR 7. It ranked for same keywords for very long time On each page from years there is the correct meta rel canonical. No spam, and WMT is ok. Could you please help? The internal links are all in http, and in https. If you are https are in https (they are all relatives) No changes have been made and the subdomain is in that way from 8 years: the main url has always been http://we.register.it Google started this indexing switch around the 15 October
Technical SEO | | luca_cash0 -
# in url affecting rank
Hi I am building links to a page www.companyname.com/category.index.php There is also another similar url www.companyname.com/category.index.php#. This page is linked to from the non # page. This is a new client and I'm not entirely sure why that link is there. Am I correct in thinking that these two urls are different in the eyes of the search engines? If so, would some of the link juice to www.companyname.com/category.index.php be transferred to www.companyname.com/category.index.php# and affect the ranking of the non # page? I hope this makes sense! Thanks
Technical SEO | | sicseo0 -
I accidentally blocked Google with Robots.txt. What next?
Last week I uploaded my site and forgot to remove the robots.txt file with this text: User-agent: * Disallow: / I dropped from page 11 on my main keywords to past page 50. I caught it 2-3 days later and have now fixed it. I re-imported my site map with Webmaster Tools and I also did a Fetch as Google through Webmaster Tools. I tweeted out my URL to hopefully get Google to crawl it faster too. Webmaster Tools no longer says that the site is experiencing outages, but when I look at my blocked URLs it still says 249 are blocked. That's actually gone up since I made the fix. In the Google search results, it still no longer has my page title and the description still says "A description for this result is not available because of this site's robots.txt – learn more." How will this affect me long-term? When will I recover my rankings? Is there anything else I can do? Thanks for your input! www.decalsforthewall.com
Technical SEO | | Webmaster1230 -
Googlebot does not obey robots.txt disallow
Hi Mozzers! We are trying to get Googlebot to steer away from our internal search results pages by adding a parameter "nocrawl=1" to facet/filter links and then robots.txt disallow all URLs containing that parameter. We implemented this late august and since that, the GWMT message "Googlebot found an extremely high number of URLs on your site", stopped coming. But today we received yet another. The weird thing is that Google gives many of our nowadays robots.txt disallowed URLs as examples of URLs that may cause us problems. What could be the reason? Best regards, Martin
Technical SEO | | TalkInThePark0 -
How to allow one directory in robots.txt
Hello, is there a way to allow a certain child directory in robots.txt but keep all others blocked? For instance, we've got external links pointing to /user/password/, but we're blocking everything under /user/. And there are too many /user/somethings/ to just block every one BUT /user/password/. I hope that makes sense... Thanks!
Technical SEO | | poolguy0 -
Is it terrible to not have robots.txt ?
I was under the impression that you really should have a robots.txt page, and not having one is pretty bad. However, hubspot (which I'm not impressed with) does not have the capability of properly implementing one. Will this hurt the site?
Technical SEO | | StandUpCubicles1