New CMS system - 100,000 old urls - use robots.txt to block?
-
Hello.
My website has recently switched to a new CMS system.
Over the last 10 years or so, we've used 3 different CMS systems on our current domain. As expected, this has resulted in lots of urls.
Up until this most recent iteration, we were unable to 301 redirect or use any page-level indexation techniques like rel 'canonical'
Using SEOmoz's tools and GWMT, I've been able to locate and redirect all pertinent, page-rank bearing, "older" urls to their new counterparts..however, according to Google Webmaster tools 'Not Found' report, there are literally over 100,000 additional urls out there it's trying to find.
My question is, is there an advantage to using robots.txt to stop search engines from looking for some of these older directories? Currently, we allow everything - only using page level robots tags to disallow where necessary.
Thanks!
-
Great stuff..thanks again for your advice..much appreciated!
-
It can be really tough to gauge the impact - it depends on how suddenly the 404s popped up, how many you're seeing (webmaster tools, for Google and Bing, is probably the best place to check) and how that number compares to your overall index. In most cases, it's a temporary problem and the engines will sort it out and de-index the 404'ed pages.
I'd just make sure that all of these 404s are intentional and none are valuable pages or occurring because of issues with the new CMS itself. It's easy to overlook something when you're talking about 100K pages, and it could be more than just a big chunk of 404s.
-
Thanks for the advice! The previous website did have a robots.txt file with a few wild cards declared. A lot of the urls I'm seeing are NOT indexed anymore and haven't been for many years.
So, I think the 'stop the bleeding' method will work, and I'll just have to proceed with investigating and applying 301s as necessary.
Any idea what kind of an impact this is having on our rankings? I submitted a valid sitemap, crawl paths are good, and major 301s are in place. We've been hit particularly hard in Bing.
Thanks!
-
I've honestly had mixed luck with using Robots.txt to block pages that have already been indexed. It tends to be unreliable at a large scale (good for prevention, poor for cures). I endorsed @Optimize, though, because if Robots.txt is your only option, it can help "stop the bleeding". Sometimes, you use the best you have.
It's a bit trickier with 404s ("Not Found"). Technically, there's nothing wrong with having 404s (and it's a very valid signal for SEO), but if you create 100,000 all at once, that can sometimes give raise red flags with Google. Some kind of mass-removal may prevent problems from Google crawling thousands of not founds all at once.
If these pages are isolated in a folder, then you can use Google Webmaster Tools to remove the entire folder (after you block it). This is MUCH faster than Robots.txt alone, but you need to make sure everything in the folder can be dumped out of the index.
-
Absolutely. Not founds and no content are a concern. This will help your ranking....
-
Thanks a lot! I should have been a little more specific..but, my exact question would be, if I move the crawlers' attention away from these 'Not Found' pages, will that benefit the indexation of the now valid pages? Are the 'Not Found's' really a concern? Will this help my indexation and/or ranking?
Thanks!
-
Loaded question without knowing exactly what you are doing.....but let me offer this advice. Stop the bleeding with robots.txt. This is the easiest way to quickly resolve that many "not found".
Then you can slowly pick away at the issue and figure out if some of the "not founds" really have content and it is sending them to the wrong area....
On a recent project we had over 200,000 additional url's "not found". We stopped the bleeding and then slowly over the course of a month, spending a couple hours a week, found another 5,000 pages of content that we redirected correctly and removed the robots....
Good luck.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Virtual URL Google not indexing?
Dear all, We have two URLs: The main URL which is crawled both by GSC and where Moz assigns our keywords is: https://andipaeditions.com/banksy/ The second one is called a virtual url by our developpers: https://andipaeditions.com/banksy/signedandunsignedprintsforsale/ This is currently not indexed by Google. We have been linking to the second URL and I am unable to see if this is passing juice/anything on to the main one /banksy/ Is it a canonical? The /banksy/ is the one that is being picked up in serps/by Moz and worry that the two similar URLs are splitting the signal. Should I redirect from the second to the first? Thank you
On-Page Optimization | | TAT1000 -
Is the URL Matching the Page Title Important?
Hello I have tried searching for an answer on this but I can't get a clear answer due to the results when searching for URL title. I have just launched our second Shopify site for one of our brands. My first site launched in 2014 but when I launched I didn't pay much heed to SEO for page titles, URLs, etc so have retrospectively fixed this over time. For my Shopify site just launching I want to get it as right as possible from the start (learning from mistakes). My question is regarding URLs and what my approach should be for better SEO. So, I have a page with a Title of Newton Leather Wallets, Purses, Card Holders & Glasses Cases and the URL is https://www.tumbleandhide.com/collections/newton-leather-wallets-card-holders It was my understanding that I should try and make the URL reflect the Page Title more accurately. The problem is that this takes the character count to 77. On other pages it can be in the 80s. Will the above link be better for SEO than say just https://www.tumbleandhide.com/collections/newton I am just wary of the URL's being too long as my Moz Site Crawl is returning a lot of URLs that are too long. Thanks in Advance.
On-Page Optimization | | lukegj0 -
Strange URL resulting a page
Hi, my friend has asked me to take a look at his site. I only know the basics of SEO so I'm learning along the way. He has some duplicate title errors showing in Moz, resulting to this page: https://www.domainname.com/about/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers/money-transfers This URL shows the 'About' page. I have tonnes of pages like this showing with really long URLs that result an actual page. Has anyone seen something like this before? I don't have a clue how this is showing the about page Any advice is greatly appreciated. Thanks James
On-Page Optimization | | Craze_Media0 -
Update old article or publish new content and redirect old post?
Hi all, I'm targetting a keyword and we used to rank quite good for it. Last couple of months traffic of that keyword (and variations) is going down a bit. I wrote an extensive new post on the same topic, much more in dept and from 600 to 1800 words covering the same topic. Is it better to update the old article and mention that it's updated recently, or publish a new post and redirect the old post to the new post?
On-Page Optimization | | jorisbrabants0 -
Advantage of using LocalBusiness rich snippets?
I am working on a website that has company profile pages. What are the advantages of using LocalBusiness rich snippets for profiles?
On-Page Optimization | | calf0 -
Sorry, but that URL is inaccessible
Our site is all on the https protocol so every time I use the on-page grader it tells me the link is unavailable. What can we do? When I use the http protocol (which is 301 redirected to the https) it still gives me the same message.
On-Page Optimization | | whiskeyfl1 -
How should I structure my product URLs?
How should I structure my product URLs for the best SEO results? Lets say my product is "American Apparel 2001". Would it be better to run the URL together or use a "-" between each word. Here are two ideas I had, but feel free to suggest others. realthread.com/products/americanapparel2001 or realthread.com/products/american-apparel-2001 Thanks for the help! Dru
On-Page Optimization | | drudalton0 -
Trailing slash on URLs
Hi everyone My question is regarding trailing URLs in Wordpress A designer setup a site and made the URL structure something like: www.website.com/description/ but I want to change it to www.website.com/description as this is more SEF A trailing / was added to the permalink structure in Wordpress, if I change this will google see these as new URLs and will all the current URLs become 404? Or should I just leave it? Thanks in advance
On-Page Optimization | | webseoservices0