Is there a tool that works like crawl test that allows more than 3000 pages?

[bradwayland]

I enjoy using crawl test inside moz but I need to find a way to crawl all the pages on a site.  It would probably be in the neighborhood of 10,000 urls.  Does anyone know of a free tool and if not is there a paid tool that will do this?

[Dr-Pete]

Just a note that we've discontinued the old Crawl Test tool and have launched an entirely new On-Demand Crawl tool based on our upgraded Site Crawl engine (launched last year). The new tool has an enhanced UI, entirely rebuilt back-end, full export capability, and will save your old crawls for up to 90 days.

We've written up a sample case study or logged-in customers can go directly to On-Demand Crawl.

If you're looking to crawl more than 3,000 pages, you can also use campaign-based Site Crawl in Moz Pro.

[MattRoney]

Hi Brad!

Donna is spot-on; I'm especially a fan of Screaming Frog. That said, it looks like you're a Moz Pro subscriber, so you can use your Moz Analytics campaigns to crawl any site up to 50,000 pages.

Whenever you set up a new campaign, it automatically crawls the site it's set to track. It then updates every week, with the crawl results surfaced in Crawl Diagnostics. It looks like your account can hold up to 5 active campaigns, but they can always be deleted and more can be added.

Does that help at all?

[DonnaDuncan]

Hi Brad,

Yes there are a couple. The two that come to mind are Screaming Frog and Xenu. Screaming frog is very popular among SEOs. The free version of will allow you to crawl up to 500 pages. A licensed version will allow you to crawl 10,000 pages or more and costs 99 pounds / year. (It's an Oxfordshire, UK company.)  Highly recommended.