Help recover lost traffic (70%) from robots.txt error.
-
Our site is a company information site with 15 million indexed pages (mostly company profiles). Recently we had an issue with a server that we replaced, and in the processes mistakenly copied the robots.txt block from the staging server to a live server. By the time we realized the error, we lost 2/3 of our indexed pages and a comparable amount of traffic. Apparently this error took place on 4/7/19, and was corrected two weeks later. We have submitted new sitemaps to Google and asked them to validate the fix approximately a week ago. Given the close to 10 million pages that need to be validated, so far we have not seen any meaningful change.
Will we ever get this traffic back? How long will it take? Any assistance will be greatly appreciated.
On another note, these indexed pages were never migrated to SSL for fear of losing traffic. If we have already lost the traffic and/or if it is going to take a long time to recover, should we migrate these pages to SSL?
Thanks,
-
Firstly, I would definitely take the opportunity to switch to SSL. A migration to SSL shouldn't be something to worry about if you set up your redirects properly, but given that most of your pages aren't indexed at all, it is even less risky.
You will eventually get the traffic back, as far as how long, it's very difficult to say.
I would concentrate on crawlability, and make sure your structure makes sense, and that you aren't linking any 404's or worse. Given the size of your site, that wouldn't be a bad thing anyway.
From your description of your pages, I'm not sure there is any "importance hierarchy", so my suggestion may not help, but you could make use of Google's API to submit pages for crawling. Unfortunately, you can only submit in batches of 100 and you are limited to 200 a day. You could, of course, prioritise or cherry pick some important pages and "hub" pages, if such things exist within your site, and then start working through those.
Following the recent Google blunder where they deindexes huge swathes of the web and, in the short term, the only way to get them back in the index was to resubmit them, someone has provided a tool to interact with the API, which you can find here: https://github.com/steve-journey-further/google-indexing-api-bulk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Web spam traffic coming from Fort Lee in Google Analytics
I keep getting a ton of web spam crawler traffic from Fort Lee. Anyone know who this is?
On-Page Optimization | | cbielich0 -
Page Analysis - Helping Product Pages Outrank Search Results Pages
Hi! We have a lot of our search results pages that have been indexed and outrank our product pages and in some instance the actual product pages barely show up at all. Here is an example query that includes our brand name: http://goo.gl/cgB6W So, we have loads of actual product pages, video pages, etc that should be showing up here, but are not and this is just one example. Unfortunately, there are a LOT of these Search Results pages out there and utlimately we would love to de-index them altogether, but it's going to have to be carefully done. So, was wondering if anyone would want to check out one of our product pages and give any feedback as to what we could change to possibly improve rank or to make them more search friendly or hopefully to help them rise above these indexed search results pages? Here is an example product page: http://goo.gl/2R4IT Thanks!! Craig
On-Page Optimization | | TheCraig0 -
Disqus Comments or IntenseDebate for eCommerce. Can It HELP???
Anybody using any of those? Do you have example of eCommerce store using it? Will it help ranking? Any thing I should know about those ''Comments'' plug-in? Thank you, BigBlaze
On-Page Optimization | | BigBlaze2050 -
Temporary redirect - help!
I ran an audit on a new clients website (they did a redesign and fell off of the radar). One of the major issues that the audit came back with was over 6,000 warnings for temporary re-directs. This is a large e-commerce site and I know there will be errors but wondering why the company who did the redesign wouldn't do a 301? Would someone be able to look at this code and tell me if what they did is correct? One of the many pages that has the temporary redirect: http://www.toyotapartsstore.com/index.php?route=checkout/cart&product_id=1 Where all the many pages with errors is redirecting to: http://www.toyotapartsstore.com/index.ph
On-Page Optimization | | KnopfBay0 -
Help With Disappearing Rankings
Hi Guys, I am stumped!!. I have been asked to look at this site http://www.quarrymotors.co.uk. Which has lost rankings for "BMW Parts" since a redesign of the site. Through a bit of detective work I have managed to get hold ot the wayback machine version of the old site here http://web.archive.org/web/20080520104847/http://www.quarrymotors.co.uk/ And according to the onpage factors I have compared with SEOquake, the keyword density, title tag, description is almost identical. I have checked webmaster tools and analytics (I only have data from June 22nd) So I am unable to confirm what traffic was available before the redesign. All other keywords:- BMW Breakers BMW Spares Are on the first page but this keyword "BMW Parts" is on page 22??!?. I have checked open site explorer and it's not a case over optimisation of anchor text as a majority of the keywords are pointing back to the url and it's one of the cleanest profiles I have ever seen. The only issue, which it can't be is right at the top left corner of the site is a piece of text "Used BMW Parts & Spares" Any help would be gracefully appreciated Kind Regards Neil
On-Page Optimization | | nezona0 -
Canonical URL tags help I am not sure what this is
I am trying to get an A grade on my webpage and this is one of the critical steps canonical URL tags I cant find much information as to what this even is never mind fixing it. Thanks I am a total newbe at this any advice is appreciated
On-Page Optimization | | gemfirez0 -
New CMS system - 100,000 old urls - use robots.txt to block?
Hello. My website has recently switched to a new CMS system. Over the last 10 years or so, we've used 3 different CMS systems on our current domain. As expected, this has resulted in lots of urls. Up until this most recent iteration, we were unable to 301 redirect or use any page-level indexation techniques like rel 'canonical' Using SEOmoz's tools and GWMT, I've been able to locate and redirect all pertinent, page-rank bearing, "older" urls to their new counterparts..however, according to Google Webmaster tools 'Not Found' report, there are literally over 100,000 additional urls out there it's trying to find. My question is, is there an advantage to using robots.txt to stop search engines from looking for some of these older directories? Currently, we allow everything - only using page level robots tags to disallow where necessary. Thanks!
On-Page Optimization | | Blenny0 -
Popup windows are coming up as 404 error in moz reports.
We have several links showing up as 404 errors because of the way our site is set up. I want to know if this is hurting our ranking because Google sees it as 404 errors or is it just something I can ignore because it works for the user? If it is hurting our quality and therefore our rankings, how can I correct it so that best practices are used? I have attached an image of the links and here is an example page --> http://www.sourcemedicalequipment.com/Perch-Polyurethane-Industrial-Stool-18-25-p/idst2.htm The links in the description section have anchor text "All 3 Choices" and result in a popup information page. If you cut and paste the URL's that they are redirecting to end in a 404 but if you use the link on the page it results in a popup information window. Hope I explained that well. Thanks for your help! wrEmQnwWQo
On-Page Optimization | | BenRWoodard0