Help Me With These Horrible Crawl Errors
-
I am currently using Opencart to run a fairly popular ecommerce website. I have done quite well in terms of rankings so far. However, the Opencart platform only goes so far.
There are endless amounts of crawl errors, from duplicate content down to missing meta tags. There is no easy way to add canonical tags. The pages that are being listed in the crawl errors seem to be mainly search URL's that i have no idea how Moz found them.
An example of the page is:
.....com/index.php?route=product/search&tag=home+office+flooring
Which has ten duplicate pages all missing meta tags.
I have used Google webmaster tools and htaccess to deny access to these pages with the slug "route=product/search" but it doesn't seem to work.
So my question to you guys is as follows:
Is it worth trying to fix these errors and do they have an effect on SEO?
If so, how do I prevent these errors that seem to grow at every crawl?
Cheers,
Danny
-
Hi Danny! How'd that crawl end up? If Andy answered your question, mind marking his response as a "Good Answer?" It'll get him some bonus MozPoints, and it helps us keep track of things in the forum.
-
I have tested with the robots tool too. Does seem to be blocked but running a crawl to see what moz brings up.
-
You don't need to do another crawl - go into WMT and go to Robots.txt tester.
Get the live robots file (if it doesn't pull it in, copy and paste it).
Find a URL you wanted blocking and see if Google crawls it and you should get a warning message
-
Great, thanks Andy, Just doing a crawl now.
-
Nope you need to add the wildcard onto the end of the product line so it looks like this
User-agent: *
Disallow: /index.php?route=checkout/cart
Disallow: /index.php?route=account/return/insert
Disallow: /index.php?route=product/*Otherwise it will just exclude that specific url.
Let me know if this works.
Thanks
Andy
-
Andy,
My robots file currently looks like this
User-agent: *
Disallow: /index.php?route=checkout/cart
Disallow: /index.php?route=account/return/insert
Disallow: /index.php?route=product/Is this correctly set up?
-
Hi
Yes its worth fixing.
Could you not add Disallow: ?route=product/* to your robots file?
Can you edit the robots file.
Thanks
Andy
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does anyone know of a predictive demographics software that helps a website predict its audience based on cookies, or whatever info it has?
Hi guys, I'm looking for a predictive analytics software for better understanding our audience online. Has anyone heard of or used a software that takes the visitors currently coming to your site and uses that data to 'predict' more information about them? Such as age, location, purchasing power, etc? Please let me know if you have!
Search Behavior | | Raconteur
Thanks,
Sabilah0 -
Do sever errors affect SERPS
hoping someone can help VPS has been causing me 10 days of server errors (im now moving off it) as my wordpress site has kept running out of memory. At the same time my traffic had dropped. Could google and bing be penalising me for number of errors, dropping me in SERPS and resulting in web traffic?
Search Behavior | | mutant20080 -
Blocking links from being crawled
For my blog, should I block category, "read more" and author links from being crawled by search engine robots? Is it good for SEO? Thanks
Search Behavior | | uesat0 -
Is putting/removing Adsense ads on the site affects crawling for seo?
Because I noticed when we totally remove Adsense ads on our site, the pages crawled per day on the google webmastertools suddenly dropped into noticable amount and we tested again to turn it on for a singe day, it jumped up again. So if Adsense affects crawling then, if we have adsense ad on all pages of the site the more chances it gets crawled by the bot? WS76R8w
Search Behavior | | CruiseControl0 -
Only 11 pages being crawled
Hi, Can some one have a look and see why out of 400+ pages we only have 11 being crawled on here?? http://www.lifetimelegal.co.uk Kind Regards Elissa
Search Behavior | | Chris__Chris0 -
404 errors
I am seeing alot of 404 erros recently but all seem to be pages like http://www.finalduties.co.uk/2011/06/21/legal-services-board-to-investigate-will-writers/feed/ (FEED) at the end of the URL also http://www.finalduties.co.uk/2011/06/23/rules-are-complex-when-it-comes-to-dying-intestate/?wpmp_switcher=desktop ( I know this wpmp_switcher is from a mobile ready plugin we used a while back which has since been removed after duplicating many pages. also ?wpmp_switcher=desktop is in my robots txt file to help block the robots from crawling these pages and displaying 404's but its still happening, Ihave 700+ 404 errors most ending in feed /tag and ?wpmp_switcher=desktop Any Ideas I know that 404's aren't that bad but seeing so many this morning I need to figure out why these are coming up all of a sudden. We have been getting these since our site was accidently taken down a while back, trying to figure out why we have lost so many rankings. we seem to have SERPS like YOYO's from 1 day to the next one keyword goes up but 150 and then next day drops same the next day???any ideas? Seems SEO posr penguin is one big contadiction, some seo experts give you one bit of advice another gives you another very confsuing. Thanks Elissa 🙂
Search Behavior | | Chris__Chris0 -
I need Help with Google!!!!
I am trying to have my picture on the first page just like SEOmoz when someone search just the name, I know have something to do with google plus, but I am so new doing that no luck or probably I am doing wrong, I have been looking in the internet but I haven't found anything. Is there anyone who can write a tutorial and post here. Maybe is already done and I don't know where. Please see the picture attached so you understand better what I want to do Again I want appear just like this but with my company http://www.sombras.co.uk/images/pic.jpg pic.jpg
Search Behavior | | teksyte0 -
Google Penalisation - Any help would be appreciated!
Hi,
Search Behavior | | ChrisHolgate
We’ve recently received a Google notification of unnatural linking along with a confirmation that we're being penalised. There were a few other sites that we owned that perhaps had too many links pointing to our main domain so we trimmed them down and submitted a reconsideration request and got the following back: "Dear site owner or webmaster of http://www.refreshcartridges.co.uk/,
We received a request from a site owner to reconsider http://www.refreshcartridges.co.uk/ for compliance with Google's Webmaster Guidelines.
We've reviewed your site and we still see links to your site that violate our quality guidelines.
Specifically, look for possibly artificial or unnatural links pointing to your site that could be intended to manipulate PageRank. Examples of unnatural linking could include buying links to pass PageRank or participating in link schemes.
We encourage you to make changes to comply with our quality guidelines. Once you've made these changes, please submit your site for reconsideration in Google's search results.
If you find unnatural links to your site that you are unable to control or remove, please provide the details in your reconsideration request.
If you have additional questions about how to resolve this issue, please see our Webmaster Help Forum for support.
Sincerely,
Google Search Quality Team" I want to stress that we have never in the past and do not currently buy any backlinks. The problem that we face now is that our site has been online for best part of a decade, there are thousands of people linking to us and I have absolutely no idea where to start. We don’t use an SEO Company but in the past few months have been using SEOmoz to improve our on-page optimisation. I know it’s a massive ask but if could a member of the SEOmoz community or a staff member quickly take a gander and let us know if anything in particular sticks out like a sore thumb it would mean a great deal to me. Of course, if needed we’ll employ the services of an SEO company but I’m hoping one of you guys will see something immediately obvious that could really help us out! Thanks in advance. Kind regards Chris0