Quickest way to deindex a large number of pages
-
Our site was recently hacked by spammers posting fake content and bringing down our servers, etc. After a few months, we finally figured out what was going on and fixed the issue. However, it turns out that Google has indexed 26K+ spammy pages and we've lost page rank and search engine rankings as a result.
What is the best and fastest way to get these pages out of Google's index?
-
Given that I'm sure you've removed these pages from your site, there will be no page to which to add a meta-noindex tag.
Disallowing these pages in robots.txt in no way signals to the search engines that they should be removed from the index, just that they should no longer be crawled. Given that they're already indexed, blocking in robots.txt would potentially save some "crawl budget" but wouldn't do anything to remove them from the index.
So submitting them to the URL Removal Tool would be by far the most effective, along with an explanation.
You'll also want to keep a very close watch on your penalty warnings within Webmaster Tools. If you get flagged, you'll want a complete history of the issue and the steps you've taken to address it in order to prepare a reinclusion request.
Lastly, don't forget to submit these same URLs to the Bing Webmaster Tools Block URLs tool. You may not get a massive amount of traffic from Bing, but there's no sense throwing it away, since you've already prepared the URL removal list anyway.
Hope that helps?
Paul
-
Yup. Just wanted to add as well that if these pages are in a particular directory, then you can deindex the entire directory in one command using the URL removal tool.
-
Disallow in robots.txt
Add a noindex meta tag to these pages
Request Google to remove the URLs from their index via WMT URL removal request
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google adding main site name to the title tags of pages in the sub folders: How to handle?
Hi community, Ours is a WP hosted website. We have given our site title which reflects across all the website page title suffix. Like "Moz SEO" will be default at the title for pages like "Local SEO - Moz SEO". We have given different page title suffix to our sub-folders' pages like blog and help guides. For blog we have given "Moz blog" as title tag suffix which was working fine. But Google suddenly started showing main website's title as suffix in pages of sub folders. Ex blog: "How to rank better - Moz blog - Moz SEO". Here we can see "Moz SEO" has been added which is not required. How to handle this? Thanks
Algorithm Updates | | vtmoz0 -
Log-in page ranking instead of homepage due to high traffic on login page! How to avoid?
Hi all, Our log-in page is ranking in SERP instead of homepage and some times both pages rank for the primary keyword we targeted. We have even dropped. I am looking for a solution for this. Three points here to consider is: Our log-in page is the most visited page and landing page on the website. Even there is the primary keyword in this page or not; same scenario continues Log-in page is the first link bots touch when they crawling any page of our website as log-in page is linked on top navigation menu If we move login page to sub-domain, will it works? I am worrying that we loose so much traffic to our website which will be taken away from log-in page sub domain Please guide with your valuable suggestions. Thanks
Algorithm Updates | | vtmoz0 -
ATTN SEO MINDS: Is there a way/tool to categorize keywords from an Omniture/GA report?
So ideally I would like to take the list of keywords I am currently ranking for, and group these based on what the user intent was in making that query. For example if I am a Thai delivery chain and I am currently receiving traffic from the queries "vegan dish" and "tofu thai food", I would want to have a column in a keyword report that says these queries fall into the VEGETARIAN category. I think what I want to know is how can I filter a massive list by a range of keywords? I want to know does this cell contain, "keyword A" or "keyword B" or "keyword Z". If so list the corresponding category. This way I can look at keyword performance by category or user intent/motivation. Is there a tool out there that will help me accomplish this, or is there a good solution in excel I can use?
Algorithm Updates | | Jonathan.Smith0 -
So, useless link exchange pages still work?!
After 3 years out of SEO I thought things might have moved on, but apparently not. Bit of back link research and all the top sites in my niche have tons of reciprocal links to barely relevant sites. Do I really have to do this? I mean I thought this was so out of date, it's not much better than keyword stuffing. So, should I just forget my lofty principles asking myself 'is this of any value to my users?' and just take the medicine?
Algorithm Updates | | Cornwall0 -
Impressions & Traffic WAY Down. Where to start?
Beginning around November 1st, I began to notice a continual, gradual drop in impressions and traffic. During the holiday season we typically see a decline in business so I initially passed it off as that, but there has been no rebound and I'm really confused on where to begin looking to figure this out. Daily impressions have now dropped from 20,000 all the way down to 5,000 and it has taken a major toll on the business (see attachment for graph of this). Some Background Information: My Site has been very static for the past 8 month's (since April '12). Admittedly Overly static with very little other than a blog post here and there added. However, during these 8 month's traffic jumped 30% so we were riding that wave and feeling confident that our past efforts built a great foundation. I'm not aware of anything even remotely black hat that has ever been done. Everything is very much on the up and up and done with the user in mind. I'm unable to track anything to a Panda update due to the consistent, gradual nature of the decline. However, with some important search queries completely falling off the map, it feels to me like we are being penalized or affected by a permanent algo change. In GWMT that are a variety of important search queries that show a change of -100%. These terms do show an average position, but when I manually search for them they are no where to be found in Google search results. This is very strange to me. It feels like we've been blacklisted for some of our more important keywords. We had a major site relaunch on January 20th (a week ago). However the downward trend was in place well before this. The site is www.mycreativeshop.com To sum it up, I'm extremely confused and very concerned with what this drop is doing to the company. I've never been in this position as we've worked very hard to lay a solid foundation and have always seen a continual, positive traffic increase. It then seemed to just start turning downward one day and won't stop. If anybody has some suggestions of how to try to get to the bottom of this and learn what is really taking place it would be greatly appreciated! Thanks, -J Wdab7Sk
Algorithm Updates | | cre80 -
Ecommerce good/bad? Showing product description on sub/category page?
Hi Mozers, I have a ecommerce furniture website, and I have been wondering for some time if showing the product descriptions on the sub/category page helps the website. If there is more content displayed on the subcategory, it should be more relevant, right? OR does it not matter, as it is duplicate content from the product page. I think showing the product descriptions on non-product pages is hurting my design/flow, but i worry that if I am to hide product content on sub/category pages my traffic will be hurt. Despite my searches I have not found an answer yet. Please take a look at my site and share your thoughts: http://www.ecustomfinishes.com/ Chris 27eVz
Algorithm Updates | | longdenc_gmail.com0 -
Any way to tell if a link has been devalued?
I have some listings in lawyer directories some of which have very hig PR , links, traffic, etc. For example, www.nolo.com, I know that Google has more or less recently devalued a lot of directory links. I would assume that a monster site like nolo would not be one of those, but does anyone know any way to tell? Paul
Algorithm Updates | | diogenes0 -
Does Google index Wordpress pages with frames
Does Google or other search engines index Wordpress pages that use frames? Here is the site in question: http://www.source-nutrition.com/son/
Algorithm Updates | | BradBorst0