Moz crawler only crawls one page?!
-
Hello there,
I'm using Moz for a while and I'm very pleased with the tool and community. But for the first time I encountered a problem. We are trying to run a crawler for a client's website but only one page (only the homepage) was crawled.
We tried to do a test on a more detailed level (maybe there is something wrong with the homepage). My campaign test's crawl came back for the Producten folder (level deeper than homepage), and it was also only a 1 page crawl with a 200 status. I did look at the robots.txt file now, and it is very restrictive, but there is nothing that I can clearly see that would explain why the crawl isn't working.
Hopefully someone can point us at the right direction.
Thanks in advance,
Jeremy
-
Hey Jeremy,
I took a look at your campaign and it looks like Lynn's suggestion that Javascript may be blocking the crawl is correct. Unfortunately, our crawler isn't sophisticated enough to parse Javascript and all of the links from the homepage are hidden within the Javascript navigation so we aren't able to find them to crawl past the homepage. I'm afraid it looks like our crawler isn't compatible with this particular site.
I apologize for the inconvenience this causes! Please let me know if I can help you with anything further.
-
Thanks!!! I will take a look at the dev blog post
-
Hi,
It can happen for a number of reasons, in my experience usually due to either unintentional blocking of the page from crawling or from being a JavaScript heavy/only site which doesn't allow content to be easily read by the moz crawler. Check out this dev blog post for a rundown on most issues and solutions. Hope it helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
On Page Grade & Page Optimization Tools not working
Hello! I have an issue with my on page grader and Page optimization. Over the past two weeks i get the attached errors. I searched the forums but after contacting our developer I am in a bit of a standstill. Could someone help on what triggers this? Would highly appreciate it. Best,
Moz Bar | | pranbg
Pran c5CSsHK c5CSsHK0 -
In Moz Campaigns, how are competitor domains tracked if they redirect their site?
Hello! One of our competitors (Company A) that we've tracked in Moz for a long time recently merged with another company (Company B) and redirected their whole site to Company B's site. Will our competitor tracking still work as-is? Or do we need to make an adjustment? I'm reluctant to delete Company A from our competitor tracking, because we will lose all of that data. But if all of the keywords are slowly going to drop off as Google starts showing Company B results only, it may be the only option. Any help is appreciated! Thanks!
Moz Bar | | PrimeFoodTeam0 -
MOZ Staff: Timeline for supporting SNI?
We have moved our blog to Amazon Web Services, and our website is soon to follow. For better or worse, AWS uses SNI, which MOZ doesn't currently support. Here are some recent forum posts about it: https://mza.seotoolninja.com/community/q/804-server-error-crawling-https https://mza.seotoolninja.com/community/q/804-https-ssl-error This makes MOZ much, much less useful to me. MOZ staff, you have a timeline for when you'll implement support for SNI?
Moz Bar | | Atomic-Object2 -
Why there is a big gap between MOZ and Google Analytics Keyword Ranking for my website?
Hello everyone, I've just started a SEO project and I'm very new in the field. Well, I tried to select the most interesting Keywords for my website. When I checked out what the current ranking for each one was, I noticed that between Google and MOZ, figures are totally different. For example, for the word "channel manager": on Google Analytics: #2 on MOZ: not in the top 50 I know that for Google Analytics, these numbers are average (12 months). But it cannot explain such a huge gap. Do you have an explanation for that? And also does that mean that I can't trust Google (really?). Because I also did some analysis with Google figures: Monthly searches, Competition... Thanks for you help! Happy Christmas, Floriane.
Moz Bar | | Frontdesk.Anywhere1 -
Deleting old obsolete pages from the Page Grades section?
Last year we did a total site overhaul, moving from an old ASP site setup to a new WordPress set up. I notice that there are still a handful of old ASP pages being graded still - and turning up as F of course. I tried marking them and deleting them, but hitting delete makes the check box clear, but does not remove the page. How do I get them totally out of grading circulation? Also, some of the links are showing with session info and extra unnecessary characters - how can I clean those links up?
Moz Bar | | OBIAnalytics1 -
Clarify "broad keyword usage in page title"
Hello Page grader has two different grades for page title that I want clarification on. There is "Broad Keyword Usage in Page Title" and "Exact Keyword Usage in Page Title". Googling around about and searching here I have found that "broad" seems to mean the keywords should be used throughout the page, rather than just in the title and header. Which makes sense as this is a kind of check to ensure the page IS about the keywords and not something unrelated. But what is meant by "broad" usage in the page title? This refers specifically to the page title and not the whole document. My best guess got me to this, given the keyword "Visit London Today"; "Come and visit London today" - exact match only "London - visit today" - broad match only "Visit London the city of dreams | visit London today" - matches both That could be complete nonsense, but basically is broad usage the use of keywords scattered in the page title? Thanks.
Moz Bar | | yolkcreative0 -
Why does Moz Analytics drop my Google Analytics connection every time I login?
I get the following message every time I check the Moz Analytics dashboard on any of the sites I manage. It seems like a bug in the system. Does anyone know how to fix it? "We lost connection with your Google Analytics account. Don't worry — you won't lose any data. Please reauthorize now."
Moz Bar | | cbizzle0 -
404 Crawl Diagnostics Report MOZ
Hi, I keep getting 404's appear in the Crawl Diagnostic error warnings. How do I find out which pages are linking to these 404 pages? How is MOZ finding them? thanks Ben
Moz Bar | | bjs20100