Help Me With These Horrible Crawl Errors
-
I am currently using Opencart to run a fairly popular ecommerce website. I have done quite well in terms of rankings so far. However, the Opencart platform only goes so far.
There are endless amounts of crawl errors, from duplicate content down to missing meta tags. There is no easy way to add canonical tags. The pages that are being listed in the crawl errors seem to be mainly search URL's that i have no idea how Moz found them.
An example of the page is:
.....com/index.php?route=product/search&tag=home+office+flooring
Which has ten duplicate pages all missing meta tags.
I have used Google webmaster tools and htaccess to deny access to these pages with the slug "route=product/search" but it doesn't seem to work.
So my question to you guys is as follows:
Is it worth trying to fix these errors and do they have an effect on SEO?
If so, how do I prevent these errors that seem to grow at every crawl?
Cheers,
Danny
-
Hi Danny! How'd that crawl end up? If Andy answered your question, mind marking his response as a "Good Answer?" It'll get him some bonus MozPoints, and it helps us keep track of things in the forum.
-
I have tested with the robots tool too. Does seem to be blocked but running a crawl to see what moz brings up.
-
You don't need to do another crawl - go into WMT and go to Robots.txt tester.
Get the live robots file (if it doesn't pull it in, copy and paste it).
Find a URL you wanted blocking and see if Google crawls it and you should get a warning message
-
Great, thanks Andy, Just doing a crawl now.
-
Nope you need to add the wildcard onto the end of the product line so it looks like this
User-agent: *
Disallow: /index.php?route=checkout/cart
Disallow: /index.php?route=account/return/insert
Disallow: /index.php?route=product/*Otherwise it will just exclude that specific url.
Let me know if this works.
Thanks
Andy
-
Andy,
My robots file currently looks like this
User-agent: *
Disallow: /index.php?route=checkout/cart
Disallow: /index.php?route=account/return/insert
Disallow: /index.php?route=product/Is this correctly set up?
-
Hi
Yes its worth fixing.
Could you not add Disallow: ?route=product/* to your robots file?
Can you edit the robots file.
Thanks
Andy
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Help with robots.txt on Magento
Hi everybody, I need your help in order to fix some problems with HTML errors and Crawling errors generated by Magento on my client's website www.casabiancheria.it I have some problems with duplicate meta informations due to the fact that there are a lot of links such as /stampe-romagnole/tovaglie-con-tovaglioli**/colore/**beige,marrone,giallo,lilla/show/all.html /stampe-romagnole/tovaglie-con-tovaglioli**/colore/**beige,marrone,lilla/show/all.html that are generated by the filter /colore/ and so they have duplicate content and meta information on them. I activated the canonicals on Magento but this hasn't fixed the problem yet. On the sitemap there are only 1 link for each product, so it seems that the canonicals are working, but bot Google Webmaster Tools and SEO Moz are giving me errors on duplicate content and meta informations. I would like to solve these problems by excluding from robots.txt all the urls that contain the filter parameters, such as /colore/, /price/, /dimensions/, etc. (take a look to the attachment). I tried different solutions in order to exclude these links from robots, but I wasn't able to succeed. Below you can find my current robots.txt... can someone help me in order to write the correct form of this file and finally exclude all these urls generated by filters on Magento? Finally, is it worth it to exclude also the images from Magento? (take a look to the final lines of the robots below). Thank you very much for your help! Alberto User-agent: *
Search Behavior | | OptimizedGroup
Disallow: /CVS
Disallow: /.svn$
Disallow: /.idea$
Disallow: /.sql$
Disallow: /.tgz$
Disallow: /w1nL1f3L0g1c/
Disallow: /app/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
Disallow: /lib/
Disallow: /pkginfo/
Disallow: /shell/
Disallow: /var/
Disallow: /404/
Disallow: /cgi-bin/
Disallow: /magento/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /api.php
Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /get.php
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /README.txt
Disallow: /RELEASE_NOTES.txt
Disallow: /?dir
Disallow: /?dir=desc
Disallow: /?dir=asc
Disallow: /?limit=all
Disallow: /?mode*
Disallow: /index.php/
Disallow: /?SID=
Disallow: /checkout/
Disallow: /onestepcheckout/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/
Disallow: /catalogsearch/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /cgi-bin/
Disallow: /cleanup.php
Disallow: /apc.php
Disallow: /memcache.php
Disallow: /phpinfo.php
Disallow: /control/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/
Disallow: /?*
Disallow: //colore/
Disallow: //price/
Disallow: //misura/
Disallow: //marca/
Disallow: //sort-by/
Disallow: //combinazione/
Disallow: /*/seleziona-colore/
Disallow: /colore/
Disallow: /price/
Disallow: /misura/
Disallow: /marca/
Disallow: /sort-by/
Disallow: /combinazione/
Disallow: /seleziona-colore/
Disallow: /*colore/
Disallow: /*price/
Disallow: /*misura/
Disallow: /*marca/
Disallow: /*sort-by/
Disallow: /*combinazione/
Disallow: /*seleziona-colore/ UmuEX4z0 -
Do sever errors affect SERPS
hoping someone can help VPS has been causing me 10 days of server errors (im now moving off it) as my wordpress site has kept running out of memory. At the same time my traffic had dropped. Could google and bing be penalising me for number of errors, dropping me in SERPS and resulting in web traffic?
Search Behavior | | mutant20080 -
Blocking links from being crawled
For my blog, should I block category, "read more" and author links from being crawled by search engine robots? Is it good for SEO? Thanks
Search Behavior | | uesat0 -
Is putting/removing Adsense ads on the site affects crawling for seo?
Because I noticed when we totally remove Adsense ads on our site, the pages crawled per day on the google webmastertools suddenly dropped into noticable amount and we tested again to turn it on for a singe day, it jumped up again. So if Adsense affects crawling then, if we have adsense ad on all pages of the site the more chances it gets crawled by the bot? WS76R8w
Search Behavior | | CruiseControl0 -
Only 11 pages being crawled
Hi, Can some one have a look and see why out of 400+ pages we only have 11 being crawled on here?? http://www.lifetimelegal.co.uk Kind Regards Elissa
Search Behavior | | Chris__Chris0 -
404 errors
I am seeing alot of 404 erros recently but all seem to be pages like http://www.finalduties.co.uk/2011/06/21/legal-services-board-to-investigate-will-writers/feed/ (FEED) at the end of the URL also http://www.finalduties.co.uk/2011/06/23/rules-are-complex-when-it-comes-to-dying-intestate/?wpmp_switcher=desktop ( I know this wpmp_switcher is from a mobile ready plugin we used a while back which has since been removed after duplicating many pages. also ?wpmp_switcher=desktop is in my robots txt file to help block the robots from crawling these pages and displaying 404's but its still happening, Ihave 700+ 404 errors most ending in feed /tag and ?wpmp_switcher=desktop Any Ideas I know that 404's aren't that bad but seeing so many this morning I need to figure out why these are coming up all of a sudden. We have been getting these since our site was accidently taken down a while back, trying to figure out why we have lost so many rankings. we seem to have SERPS like YOYO's from 1 day to the next one keyword goes up but 150 and then next day drops same the next day???any ideas? Seems SEO posr penguin is one big contadiction, some seo experts give you one bit of advice another gives you another very confsuing. Thanks Elissa 🙂
Search Behavior | | Chris__Chris0 -
I need Help with Google!!!!
I am trying to have my picture on the first page just like SEOmoz when someone search just the name, I know have something to do with google plus, but I am so new doing that no luck or probably I am doing wrong, I have been looking in the internet but I haven't found anything. Is there anyone who can write a tutorial and post here. Maybe is already done and I don't know where. Please see the picture attached so you understand better what I want to do Again I want appear just like this but with my company http://www.sombras.co.uk/images/pic.jpg pic.jpg
Search Behavior | | teksyte0