404 page not found after site migration
-
Hi,
A question from our developer.
We have an issue in Google Webmaster Tools.
A few months ago we killed off one of our e-commerce sites and set up another to replace it. The new site uses different software on a different domain. I set up a mass 301 redirect that would redirect any URLs to the new domain, so domain-one.com/product would redirect to domain-two.com/product. As it turns out, the new site doesn’t use the same URLs for products as the old one did, so I deleted the mass 301 redirect.
We’re getting a lot of URLs showing up as 404 not found in Webmaster tools. These URLs used to exist on the old site and be linked to from the old sitemap. Even URLs that are showing up as 404 recently say that they are linked to in the old sitemap. The old sitemap no longer exists and has been returning a 404 error for some time now. Normally I would set up 301 redirects for each one and mark them as fixed, but there are almost quarter of a million URLs that are returning 404 errors, and rising.
I’m sure there are some genuine problems that need sorting out in that list, but I just can’t see them under the mass of errors for pages that have been redirected from the old site. Because of this, I’m reluctant to set up a robots file that disallows all of the 404 URLs.
The old site is no longer in the index. Searching google for site:domain-one.com returns no results.
Ideally, I’d like anything that was linked from the old sitemap to be removed from webmaster tools and for Google to stop attempting to crawl those pages.
Thanks in advance.
-
I agree that the 301 redirect would be your best option as you can pass along not only users but the bots to the right page.. You may need to get a developer in to write some regular expressions to parse the incoming request and then automatically find the correct new URL. I have worked on sites with a large number of pages and using some sort of automation is the only way to go.
That said, if you simply want to kill the old URLs you can show the 404s or 410s. As you mention, then you end up with a bunch of 404 errors in GWT. I have been there too, it's like damned if you do, damned if you don't. We had some URLs that were tracking URLs from an old site and we are now here a year later (been showing 410s for over a year on the old tracking URLs) they still show up in GWT as errors.
We are trying a new solution for how to remove these URLs from the index without getting 404 errors. We show a 200 and then we put up a minimal html page with the meta robots noindex tag.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710
"When we see the noindex meta tag on a page, Google will completely drop the page from our search results, even if other pages link to it. "
So, we allow Google to find the page, get a 200 (so no 404 errors), but then use the meta noindex tag to tell Google to remove it from the index and stop crawling the page.
Remember, this is the "nuclear" option. You only want to do this to remove the pages from the Google index. Someone mentioned using GWT to remove URLs, but if I remember correctly, you only have so many pages you can do this with at a time.
If you list the files within the robots.txt. Google will not spider the files, but then if you remove the page from robots.txt file, they will start to try spidering again. I have seen Google come back a year later on URLs when I take them out of robots. This is what happened to us and so we tried just showing the 410/404, but Google still keeps crawling. We recently moved to this option with the 200/noindexmeta and it seems to be working.
Good luck!
-
You can but the 404s should stop being crawled on their own. There's a webmaster tool that you can use to make that happen faster as well
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=64033
-
Yeah it's a 404 http://www.tester.co.uk/17th-edition-equipment/multifunction-testers/fluke-1651b-multifunction-installation-tester
with over 200,000 404's its a lot to go through and 301. For some reason they it got migrated they just pointed the old url to a new one replacing the root domain name without creating matching url's. Doh.
I was thinking about robot.txt filling them all?
-
A 404 should cause Google to de-index the content. Go to one of the bad URLs and view the headers to make sure that your webserver is returning a status 404 and not just a 404 "page".
As hard and time consuming as it might be, I would still pursue a 301 option. It's the cleanest way to resolve the issue. Just start nibbling at it and you can make a dent. Doing nothing just lets the problem grow.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Copy partial content to other pages ?
One of our clients looking to redesign their website since we're redesigning the whole website we thought it would be good idea to separate services into individual pages so every service will have it's own page (currently there is 1 page that describes all of the services). what we're planing to do is to write unique content for each service page (about 300-400 keywords), but we also want to use some of the existing content which is kind of explains the process of provided services. so here i need your help! what would be the best practice to use same part of existing content on every service page without getting penalized for duplicated content? here is how we want to structure the page with h1 and h2 <main> Service name (same as page title) Subline new and unique content about 300-400 keywords Part of old content which is going to be placed on every service page </main> any help would be much appreciated!
Web Design | | MozPro30 -
Internal linking for small site
I have a site with 13 pages, 6 are product pages, 5 are free tips pages (the other 2 are the home page and contact form). Currently I have the navbar at top of site with a "products" dropdown menu for the 6 product pages and a "Tips" dropdown menu for the 5 tip pages. All categories except the contact page are at the bottom as breadcrumbs, the homepage is "home" and the rest are relevant user friendly keyword anchor text. So I have 2 more pages to ad to "Tips" and am wondering whether to have a new 2nd level tips page that links to a 3rd level of 7 different tips pages, or keep it shallow as it is, with only 2 levels from the homepage to the other (now 13) pages, with a potential of 22 pages in the foreseable few years? (and some graphics work to make it user friendly like how Zappo's has categories to the side on each of its drop down navbar menu's and non-link text categories for its bottom of page breadcrumb links) Can those aforementioned pages linking to each other in the footer dilute link equity? (I think that's one of the primary reasons I'm curious). What do you think of this: http://www.dbswebsite.com/blog/2012/08/08/internal-linking-101-5-best-practices/ (I guess I should no follow my contact page), could it be better to have a 2nd level page for "Tips" to get more equity to that page rather than across all 7 tips pages? I have read around about this on here (hence how I found out about Zappo's) and elsewhere and wanted ask to make sure.
Web Design | | Zoolander0 -
Does Google penalise for alot of advertising on your site?
I look after the search side of a decorating website on which we carry a large amount of advertising from external brands as that is our business model. Do you know if we would get penalised for having too much advertising - would it be deemed to affect the user experience? Many thanks for your help on this.
Web Design | | Pday0 -
SEO page length 4500+ words
I have read varying discussions on this... some say it is good or rather it does not really matter (as long as not stuffed with keywords) and some say more than 1000+ words is bad! I have a travel site and I want to add an historical page about the zone. It is very interesting (very organic, not written for SEO purposes as such). It adds flavor and details to a site that is really all about sales. Does anyone have an opinion whether this is detrimental to SEO or not?
Web Design | | Llanero0 -
On site SEO opinions
Hi all, I have been testing different configurations for my on-site SEO for a while now and I think I am pretty much there. However it is always nice to know what other SEO's think about my keyword density and usage. My site is http://www.tomlondonmagic.com I am curious as to what you think regarding landing page content, whether you need lots or text or little text? I have just removed links in the text as I feel I want to keep as much juice on my landing page as possible. Thanks all!
Web Design | | TomLondon0 -
Worth Splitting Up Main Site into Several Microsites?
The company I work for offers a variety of very different products, that are sold to different audiences. Right now (and for the past 4 years) all the products have been listed on one main website. Over the years, we have accumulated over 200,000 links and rank relatively well in most of the product-specific keywords. Still, for business purposes we really feel that having a unique site specific to each product would be more beneficial than having them all on one site. What are the pros and cons of making a move to different subdomains from a main site. (i.e. instead of www.cleanedison.com/solar we would set up a solar.cleanedison.com)
Web Design | | CleanEdisonInc0 -
Need help with image resizing (re: slow site)
I'm trying to figure out why I'm having speed issues with my site, and using google speed test to help me knock out some of the issues. One of issues deals with image resizing. I have a responsive design and so even though on the home page the normal width is 580 of the blog area, the full post can go up to 1170. So I size all of my images to 1170 wide and let CSS resize them depending on the size of the browser. (The images on the most recent post are a little bigger than this because I was testing something.) I was wondering what the best practice was in regard to what I'm trying to do. Also feel free to check out my site and let me know of any other feedback / advice you have. Thanks !:)
Web Design | | NoahsDad0 -
Landing pages vs internal pages.
Hey everyone I have run into a problem and would greatly appreciate anyone that could weigh in on it. I have a web client that went to an outside vendor for marketing. The client asked me to help them target some keywords and since I am new to the SEO world I have proceeded by researching the best keywords for the client. I found 6 that see excellent monthly searches. I then registered the .com and or .net domain names that match these words. I then started building landing pages that make reference to the keyword and then have links to his site to get more info. My customer sent the first of these sites to the marketer and he says I am doing things all wrong. He says rather then having landing pages like this I should just point the domain names at internal pages to the website. He also says that I should not have different looks for the landing pages from the main site and that I should have the full site menu on each landing page. I wanted to here what everyone here has to say about the pros and cons of the way to do this cause the guy giving the advice to me has a lower ranking site then I do and I have only started working on getting my site ranked this year. He has atleast according to him been doing this forever. Thanks, Ron
Web Design | | bsofttech0