Quickest way to deindex a large number of pages
-
Our site was recently hacked by spammers posting fake content and bringing down our servers, etc. After a few months, we finally figured out what was going on and fixed the issue. However, it turns out that Google has indexed 26K+ spammy pages and we've lost page rank and search engine rankings as a result.
What is the best and fastest way to get these pages out of Google's index?
-
Given that I'm sure you've removed these pages from your site, there will be no page to which to add a meta-noindex tag.
Disallowing these pages in robots.txt in no way signals to the search engines that they should be removed from the index, just that they should no longer be crawled. Given that they're already indexed, blocking in robots.txt would potentially save some "crawl budget" but wouldn't do anything to remove them from the index.
So submitting them to the URL Removal Tool would be by far the most effective, along with an explanation.
You'll also want to keep a very close watch on your penalty warnings within Webmaster Tools. If you get flagged, you'll want a complete history of the issue and the steps you've taken to address it in order to prepare a reinclusion request.
Lastly, don't forget to submit these same URLs to the Bing Webmaster Tools Block URLs tool. You may not get a massive amount of traffic from Bing, but there's no sense throwing it away, since you've already prepared the URL removal list anyway.
Hope that helps?
Paul
-
Yup. Just wanted to add as well that if these pages are in a particular directory, then you can deindex the entire directory in one command using the URL removal tool.
-
Disallow in robots.txt
Add a noindex meta tag to these pages
Request Google to remove the URLs from their index via WMT URL removal request
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page Rank on Moz compared to Ahrefs
So there seems to be a huge philosophical difference behind how Moz and Ahrefs calculates page rank (PA). On Moz, PA is very dependent on a site's DA. For instance, any new page or page with no backlinks for a 90DA site on Moz will have around 40PA. However, if a site has around 40 DA, any new page or page with no backlinks will have around 15PA PA. Now if one were to decide to get tons of backlinks to this 40 DA/15PA page, that will raise the PA of the page slightly, but it will likely never go beyond 40PA....which hints that one would rather acquire a backlink from a page on a high DA site even if that page has 0 links back to it as opposed to a backlink from a page on a low DA site with many, many backlinks to it. This is very different from how Ahrefs calculates PA. For Ahrefs, the PA of any new page or page with no backlinks to it will have a PA of around 8-10ish....no matter what the DA of the site is. When a page from a 40DA site begins acquiring a few links to it, it will quickly acquire a higher PA than a page from a 90DA site with no links to it. The big difference here is that for Ahrefs, PA for a given page is far more dependent on how many inbound links that page has. On the other hand, for Moz, PA for a given page is far more dependent on the DA of the site that page is on. If we were to trust Moz's PA calculations, SEOrs should emphasize getting links from high DA sites....whereas if we were to trust Ahref's PA calculations, SEOrs should focus less on that and more on building links to whatever page they want to rank up (even if that page is on a low DA site). So what do you guys think? Do you agree more with Moz or Ahref's valuation of PA. Is PA of a page more dependent on the DA or more dependent on it's total inbound links?
Algorithm Updates | | ButtaC1 -
Duplicate website pages indexed: Ranking dropped. Does Google checks the duplicate domain association?
Hi all, Our duplicate website which is used for testing new optimisations got indexed and we dropped in rankings. But I am not sure whether this is exact reason as it happened earlier too where I don't find much drop in rankings. Also I got replies in the past that it'll not really impact original website but duplicate website. I think this rule applies to the third party websites. But if our own domain has exact duplicate content; will Google knows that we own the website from any other way we are associated like IP addresses and servers, etc..to find the duplicate website is hosted by us? I wonder how Google treats duplicate content from third party domains and own domains. Thanks
Algorithm Updates | | vtmoz0 -
Which is the best way - to have all FAQ pages at one place, or splitted in different sections of the website?
Hi all, We have a lot of FAQ sections on our website, splitted in different places, depending on products, technologies, etc. If we want to optimize our content for Google's Featured Snippets, Voice Search and etc. - what is the best option: to combine them all in one FAQ section? or it doesn't matter for Google that this type of content is not in one place? Thank you!
Algorithm Updates | | lgrozeva0 -
Hi guys, I have a question about linking to a product page for linkbuilding. Does that count adversely vs. linking to a homepage?
Hi - so until now we have been building links via blog posts and articles and linking them to the homepage. It seems the ranking of some of my top keywords has fallen so had a few questions/concerns: Does it affect the rankings adversely if I link to the product page vs the homepage? What is rule of thumb for increasing rankings of inside pages/keywords and building links to them? Thanks
Algorithm Updates | | DGM0 -
MOZ.com Page Rank of 2?
I don't recall the page rank of SEOMoz.com prior to the company's change to MOZ.com. But did notice that MOZ.com currently has a Page Rank of 2 (which I find weird since it's such a strong, content rich, highly-regarded site). I'd be interested in hearing about findings from the MOZ.com team on why the low PR and how has it affected your site since the change? (...and perhaps a look at the future through a crystal ball 🙂 I recall reading the MOZ domain changing article titled "Domain Migrations: Surviving the "Perfect Storm" of Site Changes" which had great info and addresses some reasons for PR loss in the 'Traffic and Ranking Loss' section: http://moz.com/blog/domain-migration-lessons
Algorithm Updates | | Prospector-Plastics0 -
Too Many On-Page Links
After running a site analysis on here it has come up and said that I have a lot o pages with too many on page links and that this might be why the site is being penalized. Thing is I am not sure how to remedy this as one page that says it has 116 links is this one : http://www.whosjack.org/10-films-with-some-crazy-bitches/ Although there is only one link in the body Then again our home page has 165 http://www.whosjack.org which again it says is too many. The thing is is that surely it doesn't count on links all over the page as other wise every news homepage would be penalised? For example what would happen here on this home page? : http://www.dazeddigital.com/ Can anyone help me see what I am missing? Are there possible hidden links anywhere I should be looking for etc? Thanks
Algorithm Updates | | luwhosjack0 -
Help on Page Load Time
I'm trying to track page load time of the visits on my site and GA only says to me that it's equal to zero and page load sample is aways zero too. I've made a research, and I found that GA is used to track page load time automatically, isn't it?
Algorithm Updates | | ivan.precisodisso0 -
What is the best way to build a Mobile Site?
I am looking into building a mobile version for my website, knowing that Google Likes Google should I stick with using their mobile building tool or build something from scratch in Dreamweaver? How to optimize for Google Mobile? Thx
Algorithm Updates | | Ben-HPB0