Help recover lost traffic (70%) from robots.txt error.
-
Our site is a company information site with 15 million indexed pages (mostly company profiles). Recently we had an issue with a server that we replaced, and in the processes mistakenly copied the robots.txt block from the staging server to a live server. By the time we realized the error, we lost 2/3 of our indexed pages and a comparable amount of traffic. Apparently this error took place on 4/7/19, and was corrected two weeks later. We have submitted new sitemaps to Google and asked them to validate the fix approximately a week ago. Given the close to 10 million pages that need to be validated, so far we have not seen any meaningful change.
Will we ever get this traffic back? How long will it take? Any assistance will be greatly appreciated.
On another note, these indexed pages were never migrated to SSL for fear of losing traffic. If we have already lost the traffic and/or if it is going to take a long time to recover, should we migrate these pages to SSL?
Thanks,
-
Firstly, I would definitely take the opportunity to switch to SSL. A migration to SSL shouldn't be something to worry about if you set up your redirects properly, but given that most of your pages aren't indexed at all, it is even less risky.
You will eventually get the traffic back, as far as how long, it's very difficult to say.
I would concentrate on crawlability, and make sure your structure makes sense, and that you aren't linking any 404's or worse. Given the size of your site, that wouldn't be a bad thing anyway.
From your description of your pages, I'm not sure there is any "importance hierarchy", so my suggestion may not help, but you could make use of Google's API to submit pages for crawling. Unfortunately, you can only submit in batches of 100 and you are limited to 200 a day. You could, of course, prioritise or cherry pick some important pages and "hub" pages, if such things exist within your site, and then start working through those.
Following the recent Google blunder where they deindexes huge swathes of the web and, in the short term, the only way to get them back in the index was to resubmit them, someone has provided a tool to interact with the API, which you can find here: https://github.com/steve-journey-further/google-indexing-api-bulk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
406 Errors from Third-Parties websites In Google Webmaster Tools
Google web master tools is displaying 406 errors page.The source is not from our site domain. How to fix these issues if they are from different domain? 2KXlhRy
On-Page Optimization | | SirishaNueve0 -
Fixing Index Errors in the new Google Search Console - Help
Hi, So I have started using the new Search Console and for one of my clients, there are a few 'Index Coverage Errors'. In the old version you could simply, analyse, test and then mark any URLs as fixed - does anyone know if that is possible in the new version? There are options to validate errors but no 'mark as fixed' options. Do you need to validate the errors before you can fix them?
On-Page Optimization | | daniel-brooks0 -
Can lost pages be redirected with a 301 in order to do not lost keyword rankings?
Hi, I have got a website and some posts rank well, but I am not so interested in keep them for branding reasons. So I planned to do 301 redirects to home and remove those contents. Is it ok? Thanks!
On-Page Optimization | | patrizia_h0 -
Are there detrimental effects of having multiple robot tags
Hi All, I came across some pages on our site that have multiple robot tags, but they have the same directives. Two are identical while one is for Google only. I know there aren't any real benefits from having it set up this way, but are there any detrimental effects such as slowing down the bots crawling these pages? name="googlebot" content="index, follow, noodp"/> Thanks!
On-Page Optimization | | STP_SEO0 -
Duplicate URL errors when URL's are unique
Hi All, I'm running through MOZ analytics site crawl report and it is showing numerous duplicate URL errors, but the URLs appear to be unique. I see that the majority of the URL's are the same, but shouldn't the different brands make them unique to one another? http://www.sierratradingpost.com/clearance~1/clothing~d~5/tech-couture~b~33328/ http://www.sierratradingpost.com/clearance~1/clothing~d~5/zobha~b~3072/ Any ideas as to why these would be shown as duplicate URL errors?
On-Page Optimization | | STP_SEO0 -
How do I get rid of crawl errors?
I recently revamped all my category pages to make them Google friendly. I did a lot of keyword research and ended up deleting categories and renaming categories. Now Google Webmasters is showing a ton of 404 errors. How can I get rid of them? Do I have to remove the URL one by one with Google Webmasters? Is it something that won't hurt my rankings and will go away in time?
On-Page Optimization | | dealblogger0 -
Would removing high dynamic pages though nofollow help or hurt?
We have a sub-domain that is hosted by a third party. These pages are highly dynamic (change daily or more often) as they are product search results. Unfortunately they are raising several errors and warnings including duplicate page content, title missing or empty, long URLs, overly dynamic URL Would putting nofollows on the links to this sub-domain help, hurt or not affect page rank? As an example: Links in the middle of this page (prices) http://targetvacations.ca go to a page such as this http://travel.targetvacations.ca/cgi-bin/resultadv.cgi?id=16294922&code_ag=tgv&alias=tgv which is then redirected to a dynamic URL and presents the results.
On-Page Optimization | | TSDigital0 -
Help With Disappearing Rankings
Hi Guys, I am stumped!!. I have been asked to look at this site http://www.quarrymotors.co.uk. Which has lost rankings for "BMW Parts" since a redesign of the site. Through a bit of detective work I have managed to get hold ot the wayback machine version of the old site here http://web.archive.org/web/20080520104847/http://www.quarrymotors.co.uk/ And according to the onpage factors I have compared with SEOquake, the keyword density, title tag, description is almost identical. I have checked webmaster tools and analytics (I only have data from June 22nd) So I am unable to confirm what traffic was available before the redesign. All other keywords:- BMW Breakers BMW Spares Are on the first page but this keyword "BMW Parts" is on page 22??!?. I have checked open site explorer and it's not a case over optimisation of anchor text as a majority of the keywords are pointing back to the url and it's one of the cleanest profiles I have ever seen. The only issue, which it can't be is right at the top left corner of the site is a piece of text "Used BMW Parts & Spares" Any help would be gracefully appreciated Kind Regards Neil
On-Page Optimization | | nezona0