Xenu Alternative for Large Sites
-
We're launching a new site and we're trying to crawl it to check for any problems. It's millions of pages and Xenu seems to start encountering errors as the numbers mount past 500,000. Does anyone know of an alternative, free or paid, that could handle the size better?
-
There's a post in the last 24 hours about Xenu and Screaming Frog, it's worth a read to see if a) screaming frog my work for you and b) if the 64 bit version of Xenu mentioned in the comments can solve any problems.
http://www.seomoz.org/blog/crawler-faceoff-xenu-vs-screaming-frog
-
WEB CEO and seopowersuite try them if it doesnt work: google webmaster tools but you will have to wait until it crawls your site, and not last thing seomoz crawling it has 10 000 limit minimum but it will be enougth to detect major problems
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why is my site not being indexed?
Hi, I have performed a site:www.menshealthanswers.co.uk search on Google and none of the pages are being indexed. I do not have a "noindex" value on my robot tag This is what is in place: Any ideas? Jason
Technical SEO | | Jason_Marsh1230 -
Way to spider Wordpress site
I have an old Wordpress site and I want to move it to a new server and take it off Wordpress (too many hacks). I am trying to spider the site so as to get static, non-Wordpress, pages. I am having trouble doing this. When I spider the site, it changes the URLs. For instance, if the URL is www.domain.com/page/ the URL I get out of the spider is /page/index.html And those are not the URLs in the search engine indices. There are about 2000 pages on this site, so it is not feasible to set up 301 redirects. I tried using these spidering programs: WinHTTack Website Copier and PageNest Does anyone know of another method of turning a Wordpress site into a non Wordpress site?
Technical SEO | | DanCrean0 -
Google Not liking Magento Sites?
Hello, I'm new to the community and I wonder if anyone can help us shed a light on this SEO issue we are having. We have 3 magento websites that is being affected. Whats happening is that those site were ranked for a specific keyword for few months, but all of a sudden, it just drop like crazy. It went from top 10 to about 150 in a bout a weeks period. Some site, it's not even ranked or stopped ranking and visible on the search engine. Is google not liking MAgento for some reason?? Any help or suggestions will be appreciated! thanks
Technical SEO | | solution.advisor0 -
One large site or a few microsites?
Hi, I have a client who runs a professional expo company and wants to redo his website. Right now he has one website that has the following sections: expo company info, wedding show info, electrical wire show info, fishing show info. My question is, when I rebuild the site would it be better to do one site or four microsites?
Technical SEO | | JohnWeb120 -
Crawling a subfolder with a dev site
I am trying to set up a campaign where I am crawling a subfolder of our main site where I have dev version of the new site. However, even though the new site resolves and I have included the full resolving URL but the crawl results come back saying that only one page has been crawled. The site has had a protected block on it for a period of time but this has now been removed. Any ideas? Thanks Nick
Technical SEO | | Total_Displays0 -
International Site Links In Footer
We have several international sites and we have them linked in the footer of our main .com site . Should we add "nofollow" to these links? Our concern is that Google could see these sites as a network?
Technical SEO | | EwanFisher0 -
Best way to do a site in various regions
I have a client who has 2 primary services in 4 regions He does mold removal and water damage repair. He then serves cincinnati, dayton, columbus, and indianapolis. Before hiring my company he had like 30 domains (keyword based) and had tons and tons of fake google places listings. He actually got a lot of traffic that way. However I will not tolerate that kind of stuff and want to do things the right way. First of all what is the best site approach for this. He wants a site for each service and for each city. indy mold cincy mold dayton mold dayton water etc etc etc In the end he will have 8 sites and wants to expand into other services and regions. I feel like this is not the right way to handle this as he also has another site that is more generic To me the best way to do this is a generic domain with a locations page and a page for each city. The for the Places he would get one account - an address that is hidden since he goes to customer locations, and just multiple city defined regions. He does have an office like address at each city. So should I make him a Places listing for each city or just the one? And of course how should the actual sites be organized? Thanks
Technical SEO | | webfeatseo0 -
Problem with my site
the site is casino.pt we created the site 7-8 month ago, we started to push it by good and natural links (http://www.opensiteexplorer.org/www.casino.pt/a!links!!filter!all!!source!external!!target!page), links in sites with content rich and most of them related to gambling and sport topics. During the first 3-5 months, the rankings were better and better, after the 6 months, the site lose all its rankings. Aditional details http://www.casino.pt/robots.txt http://www.google.pt/#hl=pt-PT&source=hp&biw=1280&bih=805&q=site:http%3A%2F%2Fwww.casino.pt&aq=f&aqi=&aql=&oq=&fp=2651649a33cd228 no critical errors in google webmaster tools any idea how can I fix it? thanks
Technical SEO | | Yaron530