Mac-Friendly, DOM-Rendering Spidering Tool for Multiple Users
-
Hello!
I am looking for a spidering tool that:
- Is Mac-friendly
- Can render the DOM and find JS links
- Can spider password-protected sites (prompts for password and then continues spider, etc.)
- Has competitive pricing for 8+ users.
Screaming Frog is amazing - and maybe we're just going to have to bite the bullet there. But if anyone has any other ideas, I've love to hear them. Thanks!
-
So - after digging around a lot and reading and re-reading every article that popped up for "screaming frog alternative", I've come to the conclusion that for the price, there really is nothing better than Screaming Frog right now.
I was impressed, however, with the incredibly helpful team from Deep Crawl. This enterprise tool is designed for larger websites - whereas Screaming Frog can crap out of your local machine runs out of memory. Because it's a more powerful tool, it's more expensive than Screaming Frog - but if you need an enterprise solution, it's definitely worth looking into. Another big differentiator is that Deep Crawl has no limit to the number of users, which is our primary pain point with Screaming Frog.
-
Right now we're updating SEOSpyder ( http://www.mobiliodevelopment.com/seospyder/ ) for rendering pages but i can't give you timeframe when will be done.
So far memory requirements isn't too high and was crawl 250k site with 8G ram machine.
-
Oh actually something I just realized is that potentially ScreamingFrog can do what you want and it will provide you with access to 8 users, but the setup is complicated. You would need to run it in a big virtual machine on AWS or Google Cloud Platform. That way you can scale the machine so it won't time out and everybody will still have access to it.
Back to your question: I've worked with Deepcrawl, a bit with Ryte and more with Botify. They're all great tools that are able to crawl your site. But you probably already looked into some of them.
-
Oh, interesting - can you help me understand about more about the cloud solution are you using...? Thanks!
-
Going to follow this, as I've been looking for something too. But we went the cloud service, as there is nothing that I acme across that can otherwise fulfill all these needs.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Shopify Site with Multiple Domains?
Hey there! My client has a website on Shopify. I don't even know how to open this can of worms, but let me try. The site URL is: https://mobilityequipmentforless.com/ However, there is another (older?) URL that gets updated as the main site gets updated and shows the exact same content. It's a straight duplicate, but is it's own URL and doesn't redirect to the main site. https://www.powerchairrecyclers.com/ And this isn't the SITE.Shopify back-end site name that was used for set up initially. I just have no idea what's going on here. Not sure if it's a serious error that needs to be fixed, or if it's something weird with how Shopify work. Any insight would be immensely helpful. Thanks! Mike
Intermediate & Advanced SEO | | naturalsociety0 -
Rank Loss Analysis Tool
I did a detailed analysis of the loss in rankings for about 100 pages on my site over the last 6 months. For few pages the SEO clicks have dropped by 40%. This is not due to algorithm penalty but one of the following reason Competitor with a higher DA created a similar article Existing Competitors get more backlinks Search volume itself is down - (Don't care) Loss in long tail traffic can you guys recommend a system to automatically track this and make a report? Also what would be corrective action in case of reason 1 and 4
Intermediate & Advanced SEO | | Janki990 -
Multiple hreflang tags pointing to one page from the same country
Hi All, Hypothetically, let’s say a brand established in the UK created the following URL for the Italian market, www.example.com/it/ticket-watch (Ticket watch the made up brand) In this scenario, Ticket Watch is used across multiple services and domains in the UK such as: www.example.com/ticket-watch www.ticketwatch.com/ Essentially, could you point multiple ticket watch pages that live on different domains so that www.example.com/it/ticket-watch could potentially have 4 or 5 tags from the same country (UK), but the self-referencing pages will only have one hreflang tag: canonical and hreflang meta information to be included on www.example.com/it/ticket-watch But the hreflang meta information to be included on www.ticketwatch.com/ will only have one tag I’ve only in included 2 hreflang tags for the for the first example but let’s say there were an additional 2 or 3 GB based ticket watch hreflang tags. Will these tags still be validated? Thanks,
Intermediate & Advanced SEO | | SEONOW1230 -
Mobile Friendly Site yet rankings are in alot of cases worse than desktop
Hi All, We made our Website mobile responsive well in advance of the recent mobile algorithm changes . We only have a couple of pages with minor errors but yet our mobile rankings have not improved that much (maybe 1 place) and in alot of cases, we rank higher on desktop than we do on mobile for the same term etc. No the ranking boost is minimal at best and worse in many cases. I have checked our site on google mobile test and we are mobile friendly and on page speed test our mobile site is 99/100 user experience etc. Just wondered if anyone else has found the same ? thanks Pete
Intermediate & Advanced SEO | | PeteC120 -
One Website, Multiple Locations, One Blog?
There's definitely not going to be a "right" answer to this question, but I think it can lead to a great discussion. We are building a website for a client who has two locations, we are going to use a URL structure similar to this: www.Brand.com (this would be a landing page where users would select a location) www.Brand.com/Atlanta www.Brand.com/Boston However, we still want to focus on local SEO - so our deeper URL structure will be: www.Brand.com/Atlanta/Auto-Accident-Lawyer www.Brand.com/Atlanta/Motorcycle-Accident-Lawyer www.Brand.com/Boston/Auto-Accident-Lawyer www.Brand.com/Boston/Motorcycle-Accident-Lawyer The content on those pages will be unique and target local keywords. Each "version" of the website will have a navigation specific to that location. For example, once a user clicks into the Boston website, all of the navigation items will pertain to Boston. However, we run into an issue with the blog. Both locations will be using the same blog content, which ends up looking something like this: www.Brand.com/Atlanta/Blog/Blog-Article www.Brand.com/Boston/Blog/Blog-Article This obviously creates duplicate content. We could do something such as this: www.Brand.com/Blog/Blog-Article However, as noted above, each local version of the website has a separate navigation (this keeps a user in Boston on the Boston version of the website). So have a centralized blog is far from ideal unless navigations for both locations are included - which would allow users to return back to their local website. From my understanding, duplicate content doesn't necessarily "hurt" your SERPs, it simply keeps one of the duplicated pages from ranking. So the question comes down to this, is duplicate content a big enough issue to restructure a website to use a centralized blog?
Intermediate & Advanced SEO | | McFaddenGavender0 -
Page A Best for Users, but B Ranks
This is real estate MLS listings related. I have a page "B" with lots of unique content (MLS thumbnails mixed with guide overview writing, pictures etc) which outranks "A" which is a page simply showing MLS thumbnails with map feature included. I am linking from "B" to "A" with anchor "KEYWORD for sale" to indicate to search engines that "A" is the page I want to rank, even though "B" has more unique content. It hasn't worked so far.
Intermediate & Advanced SEO | | khi5
Questions: Should I avoid linking from "B" to "A" as that could impact how well "B" ranks? Should I leave this setup and over time hope search engines will give "A" a chance to rank? Include some unique content on "A" mostly not viewable without clicking "Read more" link? I don't foresee many users will click "Read more" as they are really just looking for the properties for sale and do rarely care about written material when searching for "KEYWORD for sale". Should I "no index, follow" A as there are limited to none unique content and this could enhance chance of ranking better for B? When I write blog posts and it includes "KEYWORD for sale" should I link to "A" (best for users) or link to "B" since that page has more potential to rank really well and still is fairly good for users? Ranking for "B" is not creating a large bounce rate, just that "A" is even better. Thank you,
Kristian0 -
Will multiple domains from the same company rank for the same keyword search?
I'm trying to convince people that we need good marketing reasons for starting multiple domains, as it will be more difficult to rank multiple sites. Does anyone know if Google actively discourages multiple domains from the same company appearing in the search results for the same keyword? We are creating a separate content website which is related to an existing company website. Would you agree that is best to have these sites on one domain with the content site on a sub-domain perhaps? I'm worried about duplication of effort and cross-keyword targeting in particular. These sites would not have duplicate content.
Intermediate & Advanced SEO | | RG_SEO0 -
Hiding Text in an SEO friendly way - is it possible?
Hello, I have a client who has very little (practically no) text content on his ecommerce website, on the home page and category / sub cat pages. We have drafted some text for him - but the designer has fought back against this as he feels it will break the design. Our proposed solution is to have some text visible - and the rest will be text that is hidden but can be revealed by clicking Read More. We are planning to follow these recommendations : http://www.shimonsandler.com/collapsible-div-seo-friendly/ We are not hiding text for the sake of it - but more to improve the UX. We of course want the text to be accessible - i.e. readable by screen readers. Does anyone have any experience or opinions in respect to taking this course of action, and is there anything we should make sure we either do or not do to stay on the side of the BIG G? Kind Regs, Rich
Intermediate & Advanced SEO | | RichBestSEO1