Recovering from Blocked Pages Debaucle
-
Hi, per this thread: http://www.seomoz.org/q/800-000-pages-blocked-by-robots We had a huge number of pages blocked by robots.txt by some dynamic file that must have integrated with our CMS somehow. In just a few weeks hundreds of thousands of pages were "blocked." This number is now going down, but instead of by the hundreds of thousands, it is going down by the hundreds and very sloooooowwwwllly. So, we really need to speed up this process. We have our sitemap we will re-submit, but I have a few questions related to it: Previously the sitemap had the <lastmod>tag set to the original date of the page. So, all of these pages have been changed since then. Any harm in doing a mass change of the <lastmod>field? It would be an accurate reflection, but I don't want it to be caught by some spam catcher. The easy thing to do would be to just set that date to now, but then they would all have the same date. Any other tips on how to get these pages "unblocked" faster? Thanks! Craig</lastmod></lastmod>
-
Hey Dan,
I am actually not so concerned about the pages being indexed. I don't really think they were ever de-indexed. Unless I am wrong, I think they were de-ranked.
I know others have said that when they "disallowed" large portions of their sites, their pages dropped in the rankings, and did not necessarily disappear. This is more what I want to see recovery from.
Thanks!
Craig
-
Craig
D'you have screaming frog? BEST way to make sure you're all set is - run a crawl with Screaming Frog. By default it will acknowledge robots.txt and not crawl anything being blocked. Set the user agent to Googlebot.
If it crawls all the pages you want it to just fine, than you are all set!
-Dan
-
Thanks for jumping in Dan. The number of blocked pages, over a month later is still way up there. It really has barely gone done. As of today it is at 904,000.
So, we still wait and hope that:
A. That many pages aren't actually blocked (whatever blocked actually means.)
B. The rate at which that number falls will begin to increase.
Thanks for your answer!
Craig
-
Hey There
I see this question is a bit old ... are you still have these issues? If so, when you say "going down" do you mean according to the numbers showing in Webmaster Tools?
I do know that quite often there can be a delay in the data in Webmaster Tools (especially the indexation report which you may be referring to).
I don't think there's any harm in updating the dates to reflect the most recent version of the page, so long as they are accurate.
Let me know if that helps or if you're all set.
-Dan
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Pricing value pages
We have the main pricing page here: https://www.eginnovations.com/product/pricing Then depending on what you click, you'll be taken to the appropriate form on one of these pages: https://www.eginnovations.com/product/request-quote?pricetype=audit https://www.eginnovations.com/product/request-quote?pricetype=saas https://www.eginnovations.com/product/request-quote?pricetype=perpetual https://www.eginnovations.com/product/request-quote?pricetype=subscription How should I handle these? Noindex, follow? Set a canonical? I keep getting notifications that these are duplicate content, but it's just a way to keep the form fills organized. Thanks for your help!
Technical SEO | | eGInnovations1 -
Any idea why pages are not being indexed?
Hi Everyone, One section on our website is not being indexed. The product pages are, but not some of the subcategories. These are very old pages, so thought it was strange. Here is an example one one: https://www.moregems.com/loose-cut-gemstones/prasiolite-loose-gemstones.html If you take a chunk of text, it is not found in Google. No issues in Bing/Yahoo, only Google. You think it takes a submission to Search Console? Jeff
Technical SEO | | vetofunk1 -
Old Content Pages
Hello we run a large sports website. Since 2009 we have been doing game previews for most games every day for all the major sports..IE NFL, CFB, NBA, MLB etc.. Most of these previews generate traffic for 1-2 days leading up to or day of the event. After that there is minimal if any traffic and over the years almost nothing to the old previews. If you do a search for any of these each time the same matchup happens Google will update its rankings and filter out any old matchups/previews with new ones. So our question is what would you do with all this old content? Is it worth just keeping? Google Indexes a majority of it? Should we prune some of the old articles? The other option we thought of and its not really practical is to create event pages where we reuse a post each time the teams meet but if there was some sort of benefit we could do it.
Technical SEO | | dueces0 -
How to identify orphan pages?
I've read that you can use Screaming Frog to identify orphan pages on your site, but I can't figure out how to do it. Can anyone help? I know that Xenu Link Sleuth works but I'm on a Mac so that's not an option for me. Or are there other ways to identify orphan pages?
Technical SEO | | MarieHaynes0 -
Page rank and ranking down
Hi I blog at Technostarry. Some 3 months back during page rank update, my page rank went down from 3 to 2. I don't know the reason behind this. And now, my traffic and ranking is also down. I am not involved in any bad SEO practices, I don't copy paste and I write original content. I am too confused as why and what has happened with my site. If someone could analyze my blog and look at my weak points then that would be great. I would like to get any suggestions to get back my ranking and also page rank back. Thanks.
Technical SEO | | technotech0 -
Too Many On Page LInk
The analysis of my site is showing that I have a problem with too many on-page links. Most of this is due to our menu, and wanting users to be able to quickly get to the shopping category they are looking for. We end up with over 200 links in order to get the menu we want. How are other people dealing with a robust menu, but avoiding getting dinged for too many links? One of our pages in question is: http://www.milosport.com/category/2176-snowboards.aspx
Technical SEO | | dantheriver0 -
Duplicate Content Home Page
Hello, I am getting Duplicate Content warning from SEOMoz for my home page: http://www.teacherprose.com http://www.teacherprose.com/index html I tried code below in .htaccess: redirect 301 /index.html http://www.teacherprose.com This caused error "too many re-directs" in browser Any thoughts? Thank You, Eric
Technical SEO | | monthelie10