Moz crawler only crawls one page?!
-
Hello there,
I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled.
We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working.
Hopefully someone can point us at the right direction.
Thanks in advance,
Jeremy
-
Hey Jeremy,
I took a look at your campaign and it looks like Lynn's suggestion that Javascript may be blocking the crawl is correct. Unfortunately, our crawler isn't sophisticated enough to parse Javascript and all of the links from the homepage are hidden within the Javascript navigation so we aren't able to find them to crawl past the homepage. I'm afraid it looks like our crawler isn't compatible with this particular site.
I apologize for the inconvenience this causes! Please let me know if I can help you with anything further.
-
Thanks!!! I will take a look at the dev blog post
-
Hi,
It can happen for a number of reasons, in my experience usually due to either unintentional blocking of the page from crawling or from being a JavaScript heavy/only site which doesn't allow content to be easily read by the moz crawler. Check out this dev blog post for a rundown on most issues and solutions. Hope it helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is the Moz on-page grader going to start grading mobile-first as Google does?
I wonder whether this has been taken into account yet or there are any plans to in future.
Moz Bar | | mybuilder1 -
Community Discussion - What's Been Your Experience With Moz Content?
When the content developed Moz Content, I was excited as can be about having another tool in the content marketing and content strategy repertoire. I knew it could and would help marketers better identify the content they should be creating and make it easier for them to move the needle for their brands. Since it's been available, I've had fun using Moz Content, seeing it as a great vehicle for flattening the learning curve for content ideation and creation. In a recent post, Here's How I'm Using Moz Content for Mining Local Link Opportunities, David Farkas described how brands can use Moz Content to better create localized content. I'd like to know how you're using it, or if you're using it: Have you tried Moz Content? And if not, what's stopping you? If you have used it, what are you really liking? What would you change? What, if any, additional features you'd like to see added? What tips can you share for helping others get the most out of the tool? Looking forward to reading the comments below.
Moz Bar | | ronell-smith3 -
804 : HTTPS (SSL) Error in Crawl Test
So I am getting this 804 Error but I have checked our Security Certificate and it looks to be just fine. In fact we have another 156 days before renewal on it. We did have some issues with this a couple months ago but it has been fixed. Now, there is a 301 from http to https and I did not start the crawl on https so I am curious if that is the issue? Just wanted to know if anybody else has seen this and if you were able to remedy it? Thanks,
Moz Bar | | DRSearchEngOpt
Chris Birkholm0 -
Keyword used 1448 times ? for a Chinese page
Hi, I tested my page grade here : https://mza.seotoolninja.com/researchtools/on_page_grader_reports/1269197 Keyword : "專案管理軟體" , which means "Project Management Software" in Traditional Chinese. URL: http://helpmecloud.com/clarizen-introduction.html But the On Page Grader told me "We found this keyword used 1448 times." I think the result is wrong, I guess you calculated all Chinese letters separately. Am I right ? How can I get a correct grade for my page ? Thansk for help! William
Moz Bar | | Helpmecloud0 -
Getting 'Sorry, but that URL is inaccessible' error msg when trying to run On-Page Grader
I just signed up for MOZ Pro for the first time today. Tried to run the 'on-page grader' tool on some of my pages but I'm getting a 'Sorry, but that URL is inaccessible' error msg. I have verified against the robot.txt file that the pages are NOT blocking any crawlers. Can anybody help?
Moz Bar | | spinoki0 -
On Page Grader Changed?
Hi, Could anyone tell me if the on page grader has changed? I remember seeing 8 boxes at the top but now i'm only seeing 6... Before I remember seeing a H2-H4 and a Strong/B Box? Thanks
Moz Bar | | dmxsta0 -
403 Error on WMT but not on MOZ?
Hello, 2 days ago I found there are about 1200 of 403 errors by Google WMT when I tried to fetch my domain - Please see attached HTTP/1.1 403 Access Forbidden Cache-Control: private Content-Type: text/html ETag: "" Server: Set-Cookie: ASPSESSIONIDSSBARTSD=BEHMJHJBKJOEJEALECNNIPFH; path=/; HttpOnly X-Powered-By: Date: Tue, 18 Feb 2014 13:54:10 GMT Content-Length: 1233 <title>403 - Forbidden: Access is denied.</title> Server Error <fieldset> 403 - Forbidden: Access is denied. You do not have permission to view this directory or page using the credentials that you supplied. </fieldset> I ran a complete report using MOZ but I was shocked not see any 4xx , 5xx errors. Google: 246 of 404 errors No Google, Yahoo or Bing blocking HTTP status code: ALL 200 301 redirect: none? I have done about 2500 over 4 years. The website is losing indexed pages. I'm not sure what's going and which numbers to trust. Please help. Thank you. Adam
Moz Bar | | homs830 -
Moz analytics not updating
Okay so I was invited to moz analytics. When I received the email I was stoked to get to use the new beta software. My campaigns transferred over ,but when I began to look at the data, it said updating check back in 24 hours or something along those lines. I thought okay that is fine, but to my suprise it has been around four days since then and it still says it is updating. It also shows weekly stats of visits but the number there is definitely wrong. It said I only had around 2,100 but I get more than that daily. Anyone in support that can help? I'm confused on what I can do to fix this issue. I understand it is just a beta ,but other people, from what I have seen, haven't had a similar issue. If anyone can point me in the right direction I'd appreciate it!
Moz Bar | | ithvac0