Mac-Friendly, DOM-Rendering Spidering Tool for Multiple Users
-
Hello!
I am looking for a spidering tool that:
- Is Mac-friendly
- Can render the DOM and find JS links
- Can spider password-protected sites (prompts for password and then continues spider, etc.)
- Has competitive pricing for 8+ users.
Screaming Frog is amazing - and maybe we're just going to have to bite the bullet there. But if anyone has any other ideas, I've love to hear them. Thanks!
-
So - after digging around a lot and reading and re-reading every article that popped up for "screaming frog alternative", I've come to the conclusion that for the price, there really is nothing better than Screaming Frog right now.
I was impressed, however, with the incredibly helpful team from Deep Crawl. This enterprise tool is designed for larger websites - whereas Screaming Frog can crap out of your local machine runs out of memory. Because it's a more powerful tool, it's more expensive than Screaming Frog - but if you need an enterprise solution, it's definitely worth looking into. Another big differentiator is that Deep Crawl has no limit to the number of users, which is our primary pain point with Screaming Frog.
-
Right now we're updating SEOSpyder ( http://www.mobiliodevelopment.com/seospyder/ ) for rendering pages but i can't give you timeframe when will be done.
So far memory requirements isn't too high and was crawl 250k site with 8G ram machine.
-
Oh actually something I just realized is that potentially ScreamingFrog can do what you want and it will provide you with access to 8 users, but the setup is complicated. You would need to run it in a big virtual machine on AWS or Google Cloud Platform. That way you can scale the machine so it won't time out and everybody will still have access to it.
Back to your question: I've worked with Deepcrawl, a bit with Ryte and more with Botify. They're all great tools that are able to crawl your site. But you probably already looked into some of them.
-
Oh, interesting - can you help me understand about more about the cloud solution are you using...? Thanks!
-
Going to follow this, as I've been looking for something too. But we went the cloud service, as there is nothing that I acme across that can otherwise fulfill all these needs.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do I need multiple 301s to preserve SEO
Many years ago I created website www.original.com. Two years ago I redirected website www.original to www.neworiginal.comusing 301 redirects. I have now created www.rebranded.com. I want to maintain all SEO value. Should I redirect both www.original.comand www.new original.com to www.rebranded.com? Or do I only need to redirect one of them and if only one which one? If I need only to redirect one can I delete the other, why or why not. Of course the url are fictitious. I truly appreciate your help
Intermediate & Advanced SEO | | PhotoStl0 -
Ecommerce New & Refurbished with multiple versions of refurbished
I am working with a website that sells new and multiple grades of refurbished power tools New Refurbished Grade A (top quality refurbished) Refurbished Grade C (had a few more scuffs but in perfect working order) Refurbished Grade D (no warranty / as is conditions, typically for parts) How would you create the Products and URL structure? Since they are all technically different products they have their own sku in magento. Would you combine them into one URL with different product options? or would you give each product version its own url (New, Grade A, Grade C, Grade D) Thanks! -- Steven
Intermediate & Advanced SEO | | intown0 -
Decreased organic traffic but increased Webmaster Tool Queries
We have a client who has had a significant decrease in organic traffic this last month (about 20%) but in Webmaster tools it tells me there was an increase in impressions and clicks. How can these both be accurate?
Intermediate & Advanced SEO | | jfeitlinger0 -
303 Redirects Search Engine Friendly?
Do 303 redirects work more like 301s or more like 302s when it comes to passing SEO authority?
Intermediate & Advanced SEO | | MiguelSalcido0 -
Best Tool For Finding Related Keywords?
What is the best tool for finding related keywords to the primary keyword we are targetting? Cheers
Intermediate & Advanced SEO | | webguru20140 -
How to add subdomains to webmaster tools?
Can anyone help with how I add a sub domain to webmaster tools? Also do I need to create a seperate sitemap for each sub domain? Any help appreciated!
Intermediate & Advanced SEO | | SamCUK1 -
How to do geo targeting for domain and sub directories in Webmaster tool?
Hello All, How can i do geo targeting in multiple countries on my ** root domain and sub **directories in Webmaster tool. My domain is "abc.com" and i want to target three countries UAE , Kuwait, Saudi arabia. So, Can i assign geo targeting in Webmaster tool , Root domain for UAE country and make other two sub directories for Kuwait and saudi ? abc.com - UAE (geo targeting) abc.com/kw - Kuwait (geo targeting) abc.com/sa - Saudi (geo targeting) Or Root doamain should be not assign for any country and Make three sub directories for UAE, Kuwait , and saudi and targeting them there geo locations. abc.com - Unlisted (geo targeting) abc.com/uae/ - UAE (geo targeting) abc.com/kw/ - Kuwait (geo targeting) abc.com/sa/ - Saudi (geo targeting)
Intermediate & Advanced SEO | | rahul110