Penalized and how to recover

[edward-may]

Hey guys

One of our sites was penalised from a while ago and we use to rank page 1 for our keywords and now w can't be found.  Is there a way to recover from this?

Also, I came across Rands video about site architecture whilst google were teaching us to generate backlinks, and spammy anchor text what would your suggestion be to recover from this?  Any ideas?

Here is the site http://free-love-psychic.com/

Any suggestions?

[MattAntonino]

It looks like your robots.txt is missing.  If you upload one and Google can read it, they will crawl your site. As the message says, further crawls of your site are "postponed" until Google can read a robots.txt file (which is odd, I've never had this on sites without robots.txt before.)

I would suggest uploading & checking your robots file and then requesting a recrawl through WMT.

[edward-may]

Hi Matt,

Going through google web masters tools, and finding the messages that were sent to me are as follows:

---

Over the last 24 hours, Googlebot encountered 30 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%.

You can see more details about these errors in Webmaster Tools.

Recommended action If the site error rate is 100%:

Using a web browser, attempt to access http://lovepsychicmary.com/robots.txt. If you are able to access it from your browser, then your site may be configured to deny access to googlebot. Check the configuration of your firewall and site to ensure that you are not denying access to googlebot. If your robots.txt is a static page, verify that your web service has proper permissions to access the file. If your robots.txt is dynamically generated, verify that the scripts that generate the robots.txt are properly configured and have permission to run. Check the logs for your website to see if your scripts are failing, and if so attempt to diagnose the cause of the failure.

If the site error rate is less than 100%:

Using Webmaster Tools, find a day with a high error rate and examine the logs for your web server for that day. Look for errors accessing robots.txt in the logs for that day and fix the causes of those errors. The most likely explanation is that your site is overloaded. Contact your hosting provider and discuss reconfiguring your web server or adding more resources to your website. If your site redirects to another hostname, another possible explanation is that a URL on your site is redirecting to a hostname whose serving of its robots.txt file is exhibiting one or more of these issues.

After you've fixed the problem, use Fetch as Google to fetch http://lovepsychicmary.com/robots.txt to verify that Googlebot can properly access your site.

So not too sure if this could be the issue?

Learn more in our Help Center.

[MattAntonino]

There are a number of different penalties so without specifics this can be hard to answer but by far the most common we see is a partial link penalty.  Basically, you built some "crap" so Google chooses to ignore it and not give you the value. Usually results in what looks like a full page, 2 page or 5+ page drop.  (Keywords that were page 1 drop to 2 or 3, keywords on 2 drop to page 5, etc.)

You would need a link clean up, maybe disavow & reconsideration request if you have this (or the full link based manual penalty.)  If you don't have a message in Webmasters Tools, you may also be talking about Penguin - an "algorithmic penalty" (not a true penalty.)  If this is what you mean, similar steps will be taken.  In either case you will also need to find a way to continue gaining authority.  Disavow / recon without authority building won't get you very far.

[waipyo]

Hi Alec, recovering from a Google penalty is not an easy process unfortunately. And the Q&A section here wouldn't be enough to explain everything as well. I would suggest going over this article on Moz. It is a very thorough article explaining every steps you need to take to remove manual penalties associated with your website. I hope this helps