Organic listings disappeared I don't know why!
-
Brief history:
I am MD of a medium sized health organisation in the UK. We have one of the leading websites in the world for our industry. We were hit by a Google algorithm update last year (Penguin or Panda, I can't remember, but that's not relevant here I don't think) and our daily visits went down from around 10,000 to around 5,000 in two separate hits over a couple of months. Then there was a steady decrease to about 3,000-4,000 visits a day until we totally updated the design of the site and did some good work on the content. We have always been white-hat and the site has around 3,000 pages with unique content added daily.
So things have really been on the up for the past couple of months. We have been receiving around 6,000 visits a day in recent weeks (a slow incline over the past few months), until Sunday. Sunday morning around 10am all of our organic listings pretty much disappear, including for our brand name. Monday morning a few come back, including our brand name and our main, most competitive keyword, which we were showing up on the third page for and we returned to this page. Then Tuesday morning another few of our most competitive keywords show up, back where they were before. This includes images which had disappeared from Google images.
Our PPC and business listings were not really affected at all.
My developer submitted a site map through webmaster tools on Monday morning and I'm not sure if this is the reason pages started to show up again. In our Webmaster tools the indexed pages are about a quarter of all of the ones on the site - all pages were indexed before. I just don't know what has happened! It doesn't make any sense as 1. Google don't seem to have rolled out any algorithm updates on that day 2. we do not have any messages in Webmaster Tools 3. a number of our main keywords have re-appeared - why would that happen if we had been hit by a Google update?!
Our organic hits, which previously made up about 80% of all our hits, have gone down by 80% and this is drastically affecting business. If this continues it is likely we will have to downsize the business and I'm not sure what to do.
When I saw that the 'indexed pages' in Webmaster tools started to increase (they were around 600 on Monday, around 900 yesterday and then this morning, around 1,300), I thought that we were on our way up and maybe this problem would just resolve itself and our listings would re-appear, but now our indexed pages have reduced slightly since this morning, back down to around 1,100 so the increase has stalled.
Can anybody help?! Do you have any idea what could be causing this? Apparently there have been no changes made to robots.txt and my developer says that no changes were made that could have affected our listings.
ANY ADVICE WOULD BE GREATLY APPRECIATED.
-
Interesting situation...and very frustrating for you, I'm sure.
You mentioned this below:
"I checked 'cached snapshot of page' in Google Toolbar for the pages that weren't being indexed, and it showed up as a 404 error. "
This sounds like you had some sort of technical error. But some things still don't add up for me. It kind of sounds like your pages were not resolving for Google. But, the odd thing is that if Google sees a 404 error, they keep trying for days, weeks or even months before they conclude that the pages should be removed from the index.
I don't have an answer for you but the first place I'd look is to make sure that your robots.txt file is not blocking Googlebot in some way. I'd also check server logs and perhaps check with your host to see if there was some significant down time for the site.
If there was a technical glitch, and the problem is now fixed, then your pages should come back into the index without you doing anything.
I'm pretty certain this isn't a penalty issue though.
-
Thank you. I will look into this, although I don't think the pages are set to no follow because there has been a further development. I checked 'cached snapshot of page' in Google Toolbar for the pages that weren't being indexed, and it showed up as a 404 error. These are pages that have always been cached before this problem occurred. I then went to 'submit URL to Google' and submitted a couple of URLs. They instantly showed up in Google's listings in the same spot as they were before and the cached snapshot of page then showed up correctly. I could do that for every page but 1. that would be a HUGE job and 2. would that look spammy or suspicious to Google? 3. is there a way of doing that for multiple pages at a time?! I feel like this problem is very close to being solved but I just don't quite know how to solve it.
-
The only time I've seen this type of thing happen - all of the pages in a site are no longer indexed, yet PPC still works, is when something on the site has been set to no-index / no-follow.
If you had a manual penalty from Google, that would show up in Google Webmaster Tools. Plus, the site would still be indexed, just ranked really, really low. If everything was missing from Google's cache, then the most likely explanation is that it was set accidentally to no-index / no-follow.
This is a very easy thing to mess up, and it's possible that someone might have hit the wrong button by accident, or updated the robots.txt file.
In the past, I had a project manager who messed this up for a client while doing a content update on the site, and it was about a week before anyone noticed. She's no longer here (not due just to that issue). But this is so critical for me and my company that we've put an automated and human testing check in place each day:
For our company, we have an automated script that runs through all of our sites (and client's sites) each day to make sure that the site is set to index / follow, both on the pages and in the robot.txt file. We also check the title tag and make sure that the name servers haven't changed.
I also pay someone on my team to run through a 12 step checklist each and every day to make sure that things like the site search are working, contact forms go through properly, and that pages are set to index / follow.
I hope this helps...
Thanks,
-- Jeff
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Decline in organic but all other sources up--should i be worried?
We migrated to a new CMS in Feb (put in redirects as needed). Since the migration, organic traffic is down 11% YOY, but direct is up 22%, FB referrals up 1,000% and PPC up 500%. Page views are up 35%. Should I be worried about the decline in organic traffic?
Intermediate & Advanced SEO | | BGR0 -
ECommerce Replatforming URL's
We are in the process of re-platforming our eCommerce site to Magento 2. For the most part, the majority of site content will remain the same. Unfortunately on our current platform, we have been inconsistent with the use of .html as a URL suffix. As a result, our category and product pages are half and half - /stainless-steel-hardware.html
Intermediate & Advanced SEO | | BoatOutfitters
&
/stainless-steel-hardware We are considering taking the opportunity to clean up and standardize our URLs. (Drop the .html from all URLs on the new site and 301 redirect these to the same URL without the .html) Our concern is that many of the .html pages are good categories with strong page rank and I've read many articles about page rank loss from 301 redirects. We are debating internally if it really makes sense to take an SEO hit for something is seemingly small as dropping the .html from the URL. It would be a no-brainer if we were taking the opportunity to change to more SEO friendly natural language URLs. However currently our URL's appear acceptable with the exception of the inconsistent suffix. Thanks in advance for any insight on how you would approach this!2 -
Stock lists - follow of nofollow?
a bit of a catch 22 position here that i could use some advice on please! We look after a few Car dealership sites that have daily (some 3 times a day) stock feeds that add and remove cars form the site, which in turn removes/creates pages for each vehicle. We all know how much search engines like sites that have content that is updated regularly but the frequency it happens on our sites means we are left with lots of indexed pages that are no longer there. now my question is should i nofollow/disallow robots on all the pages that are for the details of the vehicles meaning the list pages will still be updated daily for "new content" or allow google to index everything and manage the errors to redirect to relevant pages? is there a "best practice" way to do this or is it really personal preference?
Intermediate & Advanced SEO | | ben_dpp0 -
How can I get a list of every url of a site in Google's index?
I work on a site that has almost 20,000 urls in its site map. Google WMT claims 28,000 indexed and a search on Google shows 33,000. I'd like to find what the difference is. Is there a way to get an excel sheet with every url Google has indexed for a site? Thanks... Mike
Intermediate & Advanced SEO | | 945010 -
Organic Google Sitelinks - can I edit?
A client just contacted me saying a competitor is threatening legal action threatened against a page description in a Google sitelink (just double checked and its in the serps results too). I checked the site and the content doesn't appear on that page or anywhere else on the site. I also added a Meta description to that page to see if that will have an effect. The page is the home page and I don't really want to demote it. Is there anything else I can/should do?
Intermediate & Advanced SEO | | agua0 -
Does a 'Certified Domain' help SEO?
I see that GoDaddy offer a 'Certified Domain' option. Does this help SEO at all?
Intermediate & Advanced SEO | | Techboy0 -
Competitior 'scraped' entire site - pretty much - what to do?
I just discovered a competitor in the insurance lead generation space has completely copied my client's site's architecture, page names, titles, even the form, tweaking a word or two here or there to prevent 100% 'scraping'. We put a lot of time into the site, only to have everything 'stolen'. What can we do about this? My client is very upset. I looked into filing a 'scraper' report through Google but the slight modifications to content technically don't make it a 'scraped' site. Please advise to what course of action we can take, if any. Thanks,
Intermediate & Advanced SEO | | seagreen
Greg0