404 page not found after site migration
-
Hi,
A question from our developer.
We have an issue in Google Webmaster Tools.
A few months ago we killed off one of our e-commerce sites and set up another to replace it. The new site uses different software on a different domain. I set up a mass 301 redirect that would redirect any URLs to the new domain, so domain-one.com/product would redirect to domain-two.com/product. As it turns out, the new site doesn’t use the same URLs for products as the old one did, so I deleted the mass 301 redirect.
We’re getting a lot of URLs showing up as 404 not found in Webmaster tools. These URLs used to exist on the old site and be linked to from the old sitemap. Even URLs that are showing up as 404 recently say that they are linked to in the old sitemap. The old sitemap no longer exists and has been returning a 404 error for some time now. Normally I would set up 301 redirects for each one and mark them as fixed, but there are almost quarter of a million URLs that are returning 404 errors, and rising.
I’m sure there are some genuine problems that need sorting out in that list, but I just can’t see them under the mass of errors for pages that have been redirected from the old site. Because of this, I’m reluctant to set up a robots file that disallows all of the 404 URLs.
The old site is no longer in the index. Searching google for site:domain-one.com returns no results.
Ideally, I’d like anything that was linked from the old sitemap to be removed from webmaster tools and for Google to stop attempting to crawl those pages.
Thanks in advance.
-
I agree that the 301 redirect would be your best option as you can pass along not only users but the bots to the right page.. You may need to get a developer in to write some regular expressions to parse the incoming request and then automatically find the correct new URL. I have worked on sites with a large number of pages and using some sort of automation is the only way to go.
That said, if you simply want to kill the old URLs you can show the 404s or 410s. As you mention, then you end up with a bunch of 404 errors in GWT. I have been there too, it's like damned if you do, damned if you don't. We had some URLs that were tracking URLs from an old site and we are now here a year later (been showing 410s for over a year on the old tracking URLs) they still show up in GWT as errors.
We are trying a new solution for how to remove these URLs from the index without getting 404 errors. We show a 200 and then we put up a minimal html page with the meta robots noindex tag.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710
"When we see the noindex meta tag on a page, Google will completely drop the page from our search results, even if other pages link to it. "
So, we allow Google to find the page, get a 200 (so no 404 errors), but then use the meta noindex tag to tell Google to remove it from the index and stop crawling the page.
Remember, this is the "nuclear" option. You only want to do this to remove the pages from the Google index. Someone mentioned using GWT to remove URLs, but if I remember correctly, you only have so many pages you can do this with at a time.
If you list the files within the robots.txt. Google will not spider the files, but then if you remove the page from robots.txt file, they will start to try spidering again. I have seen Google come back a year later on URLs when I take them out of robots. This is what happened to us and so we tried just showing the 410/404, but Google still keeps crawling. We recently moved to this option with the 200/noindexmeta and it seems to be working.
Good luck!
-
You can but the 404s should stop being crawled on their own. There's a webmaster tool that you can use to make that happen faster as well
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=64033
-
Yeah it's a 404 http://www.tester.co.uk/17th-edition-equipment/multifunction-testers/fluke-1651b-multifunction-installation-tester
with over 200,000 404's its a lot to go through and 301. For some reason they it got migrated they just pointed the old url to a new one replacing the root domain name without creating matching url's. Doh.
I was thinking about robot.txt filling them all?
-
A 404 should cause Google to de-index the content. Go to one of the bad URLs and view the headers to make sure that your webserver is returning a status 404 and not just a 404 "page".
As hard and time consuming as it might be, I would still pursue a 301 option. It's the cleanest way to resolve the issue. Just start nibbling at it and you can make a dent. Doing nothing just lets the problem grow.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Log-in page ranking but not homepage
Our homepage is outranked by log-in page for "primary keyword" in Google search results; for which actually our homepage was optimised. I have gone through the other answers for the same question here. But I couldn't find them related with our website. We are not over optimised. We have link from top navigation menu of blog to our homepage. Does this causing this?
Web Design | | vtmoz1 -
Fixing my sites problem with duplicate page content
My site has a problem with duplicate page content. SEO MOZ is telling me 725 pages worth. I have looked a lot into the 301 Re direct and the Rel=canonical Tag and I have a few questions: First of all, I'm not sure which on I should use in this case. I have read that the 301 Redirect is the most popular path to take. If I take this path do I need to go in and change the URL of each of these pages or does it automatically change with in the redirect when I plug in the old URL and the new one? Also, do I need to just go to each page that SEO MOZ is telling me is a duplicate and make a redirect of that page? One thing that I am very confused about is the fact that some of these duplicates listed out are actually different pages on my site. So does this just mean the URL's are too similar to each other, and there fore need the redirect to fix them? Then on the other hand I have a log in page that says it has 50 duplicates. Would this be a case in which I would use the Canonical Tag and would put it into each duplicate so that the SE knew to go to the original file? Sorry for all of the questions in this. Thank you for any responses.
Web Design | | JoshMaxAmps0 -
Content Migration & cost of moving pages
Hope you are all having a great day! I am wondering if anyone would be able to provide general feedback. I work for a medium size company in Chicago. Currently our site is static html and we are seeking to migrate to Wordpress. After speaking with a number of website companies and receiving proposals, I am trying to understand if there is an approximate going rate or range for moving content from static html to a CMS like Wordpress? i.e. a cost per page? We don't have any dynamic content. Most of our pages are text and images. The site itself, including the blog is around 220 pages. Thanks in advance for any insight or resources!
Web Design | | SEOSponge0 -
Responsive Design vs Mobile For This Site?
They are going to do an entire website redesign for http://gaport.com/ and I think they should adhere to responsive design best practices. However, I'm just saying that because everything I have read says that's the "way of the future" if not the way of the present already. Any reason, they shouldn't do that and keep the desktop/mobile sites? Thanks, Ruben
Web Design | | KempRugeLawGroup0 -
Site Doing Horrible After Redesign
Hello Fellow Forum Members: Thank you all for taking the time to read this. This is in follow up to one of my previous questions, but I now have more information. I will try to be as concise as possible and want to sincerely thank anybody who invests time in answering this. Around February 9, 2013, we launched our new site on the Bigcommerce platform. We moved from Volusion after 6 years. We had paid the Bigcommerce partner for an upgraded 301 redirect package as I was thoroughly concerned about losing rankings. By the end of February our rankings were diminishing. We expected a slight dip due to the new site. As of May, our organic traffic had dropped by 82%. Google WMT is showing 1500+ 404 errors. Many have to do with review page type URLs and some were just plain never redirected apparently. In May, we hired a wonderful SEO company that is a heavy contributor to the Moz community. They have been generous and wonderful to work with. By the end of this last week it was determined that most of the coding suggestions our SEO was making could NOT be implemented in Bigcommerce because Bigcommerce will not allow access to the PHP files by our developer, thus hindering the execution of these suggestions. Some of these were move the blog to the root, use canonical on the home page, use canonical for pagination, stop the indexing of https URLs and a few more. Today, June 25 we are at a complete loss and trying to just keep our business alive. The opinion of both the SEO and the developer is that my choice of Bigcommerce as a platform was not the best. So my main question is what are the odds our rankings have decreased due to the lack of 301 redirects during our migration to Bigcommerce versus the rankings decreasing do to Bigcommerce being a bad choice as a platform? We are being advised to redevelop our entire site on an Open Source platorm such as Wordpress or Magento, but if that's not needed I certainly don't want to have to do that. I hope I have provided a decent amount of history and information. Thank you for any help/advice you are willing to offer.
Web Design | | josh3300 -
Local Versions of Pages
I have a site that offers services across two states and was wondering if I would see any benefit from creating pages such as: SERVICE in CITY, TX Would I need to change the content on the pages completely or could I simply swap out the city/state if I have roughly 3-5 combos I want target?
Web Design | | nusani0 -
How can I reduce my warnings for excesive links on our site?
Our campaign overview shows well over 100 warnings that could be hurting our google ranking based on excessive links on pages. Each page listed, however, is simply due to listing the brands we carry, and linking to the products. Is there a way to do this without hurting our ranking? A better way than linking, perhaps? Thanks in advance!
Web Design | | guycochran0 -
Ecommerce web site with too many internal links
Hi, We're using Magento CE 1.4.0.1 for our ecommerce web site with a fairly flat navigation system i.e. 9 major categories display across the top menu that when you roll over display 2-20 sub categories (which take you to a groups of similar products) and then individual product pages. The categories and sub categories are available to click on as part of a dynamic Html menu system on each page. Each page also shows a small number of related products. This linking structure seems fairly standard and yet Seomoz throws up the error message, "Too Many On-page links" for most pages on our site. Do I need to really worry about this? Is there much can be done to improve this on an ecommerce web site with a large catalogue of products? I've looked at the Knowledge Base but I don't feel the existing responses adequately address the issue for ecommerce sites.
Web Design | | languedoc0