MOZ Crawler
-
Hi,
how much time it will take MOZ crawler to take entire site? In 24 hours it crawled only 500 pages isn't it too slow? My website has almost 50k pages.
-
Hi there,
Sam from Moz's Help Team here! Could you pop a message over to [email protected] about this so we can ensure everything is running smoothly for you?
Thank you!
-
Yeah it will take a while but it's necessarily slow. Moz's crawler has to share bandwidth with your real, human users. If they crawl your site too fast they could lag your site or accidentally DDoS you or something
Since it's a paid for service, I'm guessing that Moz err on the side of caution with Rogerbot and don't crawl insanely fast. I know that Rogerbot supports adding a crawl delay to make the crawling even slower:
https://mza.bundledseo.com/help/moz-procedures/crawlers/rogerbot
But I don't think there are any 'go faster' commands. Whilst I really like Moz's crawler I tend to find that the SEMRush one is faster for on-site stuff, though it does have URL crawling caps
To be honest, just get Screaming Frog and learn how to use that. Now that it has a database storage mode (instead of maniacally trying to store all the crawl data in the RAM) - it probably could crawl all 50K URLs. It could probably do it faster than Rogerbot, though you wouldn't wanna go nutty as you could easily DDoS yourself :')
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why is MOZ crawl is returning URLs with variable results showing Missing Meta Desc? Example: http://nw-naturals.net/?page_number_0=47
Can you help me dive down into my website guts to find out why the MOZ crawl is returning URLs with variable results? And saying this is missing a description when it's not really a page? Example: http://nw-naturals.net/?page_number_0=47. I've asked MOZ but it's a web development issue so they can't help me with it. Has anyone had an issue with this on their website? Thank you!
Moz Pro | | lewisdesign0 -
How to set up Goals/Conversions in Google Analytics with Moz
With my new Moz account, how would I go about setting up a useful Goal/Conversion on Google Analytics?
Moz Pro | | Brandon320 -
Possible Crawling Problem with Screaming Frog and Moz Crawlers
So I'm not sure if what I'm seeing is a problem or not. As of about two weeks ago the Moz crawler has only been able to see www.mysite.com, and none of the links, content, title, ect associated with the page. Essentially the report has one line, what should be the homepage, but it's not able to pull any information from the page but does show a 200 http status code. The report shows nothing blocked by robots or any errors. When I use screaming frog to crawl the site about 75% of the time it just reports one line www.mysite.com with a 200 status code, but again the crawler is not able to actually see the html. The other 25% of the time it works perfectly fine, crawls all pages and sees all meta info and content. There are no errors in Google WMT and everything looks ok there. We have seen a traffic drop the last two weeks but I don't know if this is the reason for it. I can't publicly post the page but if someone has an idea of what might be going on I'd be happy to PM them. Thanks
Moz Pro | | CJ50 -
Link Analysis-Moz analytics
Good afternoon, I relay hope someone can help me out with this report In link analysis I have viewed inbound links where I see many links linking to our site but among them I see also many times our site listed with various anchor text. How is this possible. Our site is linking to our site or what? It realy does not make sense to me. Can somebody explain me this please?
Moz Pro | | Rebeca10 -
How to handle subdirectories in Moz and Webmaster Tools?
I have 3 websites: http://www.abc.com (English pages) http://www.abc.com/fr (French) http://www.abc.com/de (German) Webmaster Tools In Google Analytics: I created an Account called ABC and created 4 properties with relative filters ABC Default (No filter) ABC English (filter to exclude /fr, /de) ABC French (filter to include /fr) ABC German (filter to include /de) In webmaster tools, I wish to add 3 sites http://www.abc.com http://www.abc.com/fr http://www.abc.com/de How do I validate each url with Google analytics, considering I just want the specific language for folder. How do I validate the domain name http://www.abc.com to always ensure the inclusion of the default english data? MOZ Do I need to create 3 campaigns (one per language) and if yes, again how do I handle http://www.abc.com which should exclude /fr and /de? Thank you
Moz Pro | | seo12120 -
Are Moz Ratings that reliable/correlated against SERPS?
So 3 weeks in Penguin 2.0 and after an initial drop in rankings (4-5 places on top keywords from rank 1), I have seen a small climb in rankings by a couple places but have seen a huge jump in my DA, PA and Domain trust/rank... That FAR surpasses any other magicians website. However I am ranking below some really really poorly rated sites. I am getting a PA of 51 which is really high for the industry (maybe the highest). If this is the case, why am I not number one for every keyword in my demographic?
Moz Pro | | TomLondon
I know this is asked a lot but how reliable are Moz stats, I know no one can really know how Google rank sites on SERPS but if Moz is the best shouldn't it be kinda close? Any thoughts on how accurate Moz rank is and also if it can be used as some indication of the SEO power behind a website.0 -
Big drop in Moz Domain Authority in the latest mozscape update
hi, Before Moz were forced to roll back the moz scape I saw a drop in all 3 of my campaigns domain authority; 42 > 39
Moz Pro | | FDFPres
39 > 35
24 > 22 I am employing differing tactics on these campaigns, one of which i would say leans towards the less white (the one in the middle of those stats). The other 2 however are all clean. So, did anyone else see a drop in DA before moz retracted the figures? was there an issue with the latest moz update? Thanks in advance.0 -
How can i find competitor title and descriptions in SEO moz PRO
Hi I can't seem to find competitor title and descriptions in SEO moz PRO. I am missing something?
Moz Pro | | smashseo0