Is there a tool that works like crawl test that allows more than 3000 pages?
-
I enjoy using crawl test inside moz but I need to find a way to crawl all the pages on a site. It would probably be in the neighborhood of 10,000 urls. Does anyone know of a free tool and if not is there a paid tool that will do this?
-
Just a note that we've discontinued the old Crawl Test tool and have launched an entirely new On-Demand Crawl tool based on our upgraded Site Crawl engine (launched last year). The new tool has an enhanced UI, entirely rebuilt back-end, full export capability, and will save your old crawls for up to 90 days.
We've written up a sample case study or logged-in customers can go directly to On-Demand Crawl.
If you're looking to crawl more than 3,000 pages, you can also use campaign-based Site Crawl in Moz Pro.
-
Hi Brad!
Donna is spot-on; I'm especially a fan of Screaming Frog. That said, it looks like you're a Moz Pro subscriber, so you can use your Moz Analytics campaigns to crawl any site up to 50,000 pages.
Whenever you set up a new campaign, it automatically crawls the site it's set to track. It then updates every week, with the crawl results surfaced in Crawl Diagnostics. It looks like your account can hold up to 5 active campaigns, but they can always be deleted and more can be added.
Does that help at all?
-
Hi Brad,
Yes there are a couple. The two that come to mind are Screaming Frog and Xenu. Screaming frog is very popular among SEOs. The free version of will allow you to crawl up to 500 pages. A licensed version will allow you to crawl 10,000 pages or more and costs 99 pounds / year. (It's an Oxfordshire, UK company.) Highly recommended.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Top pages > PNGs, JPGs & PDFs > What to do?
After exporting Top pages list in Moz Pro, I noticed lots of pages with high PA (page authority) are actually random images (.png & .jpg) and documents (.pdf). Question 1: Should I make a redirect 301 from an image/document to the actual webpage? If not, what should I do to keep the ranking, but point the URL to the correct place? In addition, lots of highly ranking URLs are actually outdated. They are coming from the old website. I switched to the new one more than a year ago. Question 2: If I create a 301 redirect from the old URL to the new one, will I keep the high ranking? Thanks,
Moz Bar | | Flowrox
Anja0 -
Blocked Resource in Google Index. SSL certificate blocking 718 pages seen in Google Search Console.
My google search console indicates that my SSL certificate is blocking Googlebot. I was wondering if the blocking of my SSL certificate to the GoogleBot is causing any issues. I I'm not sure if this was only blocked recently by Volusion (my host) as a means of accommodating my ssl certificate not being able to address the various url versions of my site, or is this just commonplace and not really harmful to my indexing. I tested one of these "blocked" urls in the robots.txt tester and it showed that the Googlebot was allowed. Could it be just the SSL certificate at the bottom of the page is blocked? Thanks
Moz Bar | | mrkingsley0 -
Significant difference with DA scores on Moz Chrome app VS Website Tool
I saw a big difference between DA scores on Moz Chrome app VS Website Tool. Which DA score is the correct one? I personally believe the scores on Moz's site is most accurate. Do you happen to have issues syncing the scored to your Chrome app? Chrome App VS. Moz Research Tool (https://mza.seotoolninja.com/researchtools/ose/)
Moz Bar | | iPrice_Marketing
Moz.com: 92 / 88 Here's a screenshot of the difference: https://ipricegroup-my.sharepoint.com/:i:/p/jeremy_chew/ESy9lzUTC3lOl8o93Sx6LsUB8mYXo8LmYuUj2aa0xLXi1A?e=A6OrXs This was evident in other websites too:
Priceza (priceza.com.my😞 38 / 24
Shopback (shopback.my😞 43 / 41
Cuponation (https://www.cuponation.com.my/😞 27 / 250 -
Mozbar SERP overlay stopped working in Firefox
Hello, I recently upgraded Firefox and the Mozbar SERP overlay stopped working. I rely on that data so need to get it working. Are others experiencing this issue?
Moz Bar | | IrvCo_Interactive0 -
Way has the number of pages crawled plummeted?
Why has the number of pages crawled for our campaign plummeted in Moz Analytics – down to 729 from over 10k? Don't see any issues in Google Analytics with crawling our site.
Moz Bar | | EyeglassesGuy0 -
Delete On Page Reports, Modify Keywords
Hi, I tried modifying the incorrect keyword assignments for my pages but there is no pencil icon. I also tried deleting the incorrect reports and redoing them but the incorrect OP reports are not deleted and the new reports are not saved when I refresh I tried following the instructions here http://moz.com/help/guides/search-overview/on-page-optimization/manage-page-grades but they didn't work. Thanks
Moz Bar | | xvpn9020 -
Since the revised website was launched, I can't find the "Crawl Test" function showing Titles and Descriptions of other websites. Anyone know where that link is located?
MOZ can "crawl" any website and show information like Title, Description, etc.....Can't find that link.
Moz Bar | | bpedrazas0