Moz not crawling opencart product pages
-
hi,
i have waited for over 2 weeks now and the crawler only got 8 pages, and is not getting all the open cart pages and products.
any idea of what can be wrong?
im using joomla 2.5.11 and mijoshop 2.0.5 (which uses opencart 1.5.5.1).
thanks
-
Hi Marlon,
There are a lot of different issues that may cause Rogerbot to be unable to crawl your site. For troubleshooting tips, I suggest reading http://moz.com/help/pro/why-can-t-rogerbot-crawl-my-site. Hope that helps!
-
ok thanks
-
Screaming Frog has an option in the menu -
Configuration > Spider > "Ignore Robots.txt"
Tick this box and re-scan and see if you get different results.
And also, may be worth checking this out:
http://moz.com/learn/seo/robotstxtGood luck!
-
hi
thanks for the answer, tried screaming frog now as well and seems it has the same problem,
is it possible to allow only certain spiders/robots to crawl all files, with the robots.txt?
thanks
-
It's hard to say without knowing the domain name -
Have you tried Screaming Frog software to see if it also has issues crawling?
Double check your robots.txt to see if some directories and pages are blocked.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Pro Question: Does the amount of keywords you are allowed to search reset each month?
I am a Moz Pro subscriber and I really love the new Keyword Explorer tool. One question I have that I couldn't find a clear answer was regarding the number of monthly keyword queries. Do they reset each month? I hope they do.
Moz Bar | | joemaclean0 -
MOZ Freshweb reported mentions that really does not contain any mention - anyone else experiencing this?
MOZ Freshweb reported links from "traffic.fullcontentrss.com" that doesn't really contain any mention. Is this a MOZ error or is it maybe since fullcontentrss.com has a javascript that pulls data into their own RSS, MOZ ended up crawling partially and as a result not reporting the correct URL? Anyone else experiencing this anomaly?
Moz Bar | | CPR_PTANTONO0 -
Community Discussion - What's Been Your Experience With Moz Content?
When the content developed Moz Content, I was excited as can be about having another tool in the content marketing and content strategy repertoire. I knew it could and would help marketers better identify the content they should be creating and make it easier for them to move the needle for their brands. Since it's been available, I've had fun using Moz Content, seeing it as a great vehicle for flattening the learning curve for content ideation and creation. In a recent post, Here's How I'm Using Moz Content for Mining Local Link Opportunities, David Farkas described how brands can use Moz Content to better create localized content. I'd like to know how you're using it, or if you're using it: Have you tried Moz Content? And if not, what's stopping you? If you have used it, what are you really liking? What would you change? What, if any, additional features you'd like to see added? What tips can you share for helping others get the most out of the tool? Looking forward to reading the comments below.
Moz Bar | | ronell-smith3 -
Moz is finding phantom pages
I suddenly have 4xx errors in my crawl diagnostics because pages with “/%3C/div” added to the end of the URL that are linked from the normal page can't be found. I didn't create the pages, and they don't exist, but Moz thinks that they do. I went back through to see if any changes in WordPress, theme or plugins versions might be the cause, but this is the only site that I have this issue, so I don't think that is it. Does anyone have an idea what causes this?
Moz Bar | | samuelldrew0 -
Moz Crawl Test Trying to Crawl Contact Form Submit Button Location?
Moz Crawl Test for some reason is trying to Crawl a contact form Widget Submit Location. My obvious guess is that obviously the crawl cannot submit to the required fields…..I believe this because they're only kicking back these errors on the pages I have a contact form widget on. http://crawfordspest.com/pest-control/[email protected] 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404
Moz Bar | | Funk-Creative-Media
http://crawfordspest.com/tree-services/[email protected] 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404
http://crawfordspest.com/lawn-care/[email protected] 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404
http://crawfordspest.com/specialty-services/[email protected] 1412553693 404 : Received 404 (Not Found) error response for page. Error attempting to request page; see title for details. 404 Can you shed any insight to this? I'm a bit worried that I'll have to complete gut the contact form which was one of the major requests my client requested. Or in a worse scenario make all fields not required. It would let so much spam in. I have never seem anything like this at all. But I've learned a lot from Moz, and with major errors like 404 damage Domain Authority greatly. I've fixed 404 issues with newly acquired clients existing sites and tracked through Moz and the domain authority flies up once these errors are fixed. Along with fixing what Webmaster Tools through Google reports back. ..... Let me know if you have any expertise on this matter.0 -
Moz toolbar Broken ?
Hi everybody, It seems the Moz toolbar isn't working properly for two days (Amsterdam Time zone GWMT +1) - we are located in the Netherlands. My colleague and me are have the same experience. Nothing is displayed anymore (no PA, mR, DA or Links) the sections with the pencil (follow and no follow) is working. Working in firefox 26.0.Does anybody has the same issue? Moz is this a global / local problem?
Moz Bar | | lveembergen0 -
Way has the number of pages crawled plummeted?
Why has the number of pages crawled for our campaign plummeted in Moz Analytics – down to 729 from over 10k? Don't see any issues in Google Analytics with crawling our site.
Moz Bar | | EyeglassesGuy0 -
Moz Analytics/Reports into PDF
Hey guys The Moz Analytics is great however I don't seem to be able to download the data into a PDF. I used to be able to do this but it won't let me anymore. Its vital we can do this so we can send to our clients anybody got any ideas? Am I being blind or has this been omitted??
Moz Bar | | tempowebdesign1