Handling a Huge Amount of Crawl Errors
-
HI all,
I am faced with a crawl errors issue of a huge site (>1MiO pages) for which I am doing On-page Audit.
-
404 Erorrs: >80'000
-
Soft 404 Errors: 300
-
500 Errors: 1600
All of the above reported in GWT.
Many of the error links are simply not present on the pages "linked from". I investigated a sample of pages (and their source) looking for the error links footprints and yet nothing.
What would be the right way to address this issue from SEO perspective, anyway? Clearly. I am not able to investigate the reasons since I am seeing what is generated as HTML and NOT seeing what's behind.
So my question is: Generally, what is the appropriate way of handling this?
-
Telling the client that he has to investigate that (I gave my best to at least report the errors)
-
Engaging my firm further and get a developer from my side to investigate?
Thanks in advance!!
-
-
Usually an on page audit lists all of the problems and possible reasons why they are happening, not in depth info on how to fix all the issues. That is usually the next phase, "do you want me to work on the site or do you want your dev team to track down the cause of the issues and fix them"
It also depends what type of contract you have with him of course.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website server errors
I launched a new website at www.cheaptubes.com and had recovered my search engine rankings as well after penguin & panda devestation. I'm was continuing to improve the site Sept 26th by adding caching of images and W3 cache but moz analytics is now saying I went from 288 medium issues to over 600 and i see the warning "45% of site pages served 302 redirects during the last crawl". I'm not sure how to fix this? I'm on WP using Yoast SEO so all the 301's I did are 301's not 302's. I do have SSL, could it be Http vs Https? I've asked this question before and two very nice people replied with suggestions which I tried to implement but couldn't, i got the WP white screen of death several times. They suggested the code below. Does anyone know how to implement this code or some other way to reduce the errors I'm getting? I've asked this at stackoverflow with no responses. "you have a lot of http & https issues so you should fix these with a bit of .htaccess code, as below. RewriteEngine On
On-Page Optimization | | cheaptubes
RewriteCond %{HTTPS} !=on
RewriteRule ^.*$ https://%{SERVER_NAME}%{REQUEST_URI} [R,L] You also have some non-www to www issues. You can fix these in .htaccess at the same time... RewriteCond %{HTTP_HOST} !^www.
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}/$1 [R=301,L] You should find this fixes a lot of your issues. Also check in your Wordpress general settings that the site is set to www.cheaptubes.com for both instances." When I tried to do as they suggested it gave me an internal server error. Please see the code below from .htaccess and the server error. I took it out for now. BEGIN WordPress <ifmodule mod_rewrite.c="">RewriteEngine On
RewriteBase /
RewriteRule ^index.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
RewriteEngine On RewriteCond %{HTTPS} !=on RewriteRule ^.$ https://%{SERVER_NAME}%{REQUEST_URI} [R,L]
RewriteCond %{HTTP_HOST} !^www. RewriteRule ^(.)$ http://www.%{HTTP_HOST}/$1 [R=301,L]</ifmodule> END WordPress Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, [email protected] and inform them of the time the error occurred, and anything you might have done that may have caused the error. More information about this error may be available in the server error log. Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request.0 -
How to fix Medium Priority Issues by mozpro crawled report??
How to resolve this issues crawled by mozpro?
On-Page Optimization | | renukishor
Some Medium priority issues like that: Missing Meta Description Tag: 2669
Title Element is Too Long: 523
Duplicate Page Title: 37 How to add missing meta description tag in these pages and how to short title element ?0 -
How can I fix multiple 404 errors with Wildcard htaccess redirect
Hi all I hope that someone can help.... How can I fix multiple 404 errors with Wildcard htaccess redirect The url in question is: How can I fix multiple 404 errors with Wildcard htaccess redirect http://www.5starweddingdirectory.com/listing/search/Category/luxury_hotels_venues_uk_wedding_venues/exclusive_use_venues/letter/c http://www.5starweddingdirectory.com/listing/location/uk-england/bedfordshire-weddings/franklin-park http://www.5starweddingdirectory.com/deal/location/uk-england/chorley-weddings/curtis-bay etc, going to http://www.5starweddingdirectory.com/business the above is just a few examples, google webmaster is showing over 8.000 404 page not found errors. Thanks in advance.
On-Page Optimization | | Taiger0 -
I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?
Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?
On-Page Optimization | | absoauto0 -
Errors when checking W3C HTML after added Google Custom Search
hello, I have added google custome search to my website, and then check with W3C HTML, it report many error.
On-Page Optimization | | JohnHuynh
eg: there is no attribute "enableHistory" <gcse:searchbox-only enablehistory="true" autocompletemaxcompletions="5" au…<br="">or there is no attribute "resultsUrl" and so on ...</gcse:searchbox-only> Has anyone face with this problem, I don't know how to fix it. Please help!0 -
How should I handle author attribution for ghostwritten content?
I've been using Crowdcontent for article production, and always feel like I'm potentially missing out on some authority or social proof with visitors (and maybe Google?) by not attributing an author (Crowdcontent doesn't give you the name of the author, otherwise I would just use their name). Would I be doing myself any favors by attributing myself as the author and pointing it back to my Google+ profile? Thanks in advance for any guidance!
On-Page Optimization | | spking0 -
How does a keyword get crawled and pointed at a certain page
I was wondering if you can give me some insight on how a keyword that I put on my campaign gets linked to a specific URL on my website by SEOMoz or Google. For example: updating a brick fireplace is my keyword. On the campaign when I am looking at my on page optimization, the URL assigned (or given) to it is my homepage. How is this determined and is there a way around it and or directing it to the correct page? Thanks
On-Page Optimization | | SammyT0 -
Should a crawl ever take more than 7 days?
I signed up to the 30 day trial last Saturday, however, as of yet, the crawl diagnostics page still says 'First crawl in progress'. Is this normal? Many thanks.
On-Page Optimization | | danzspas0