Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
-
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS.
Thanks
-
Hey Stephen,
THanks - I have heard of Xenu many times before. I'll try that out.
I appreciate the input!
-
just going to throw in an alternative for this type of thing Xenu Link sleuth - free, easy and powerful.
Yes that site looks like ass but the tool is great. Dr Pete and others have done "how to use" posts on seomoz - just search xenu on seomoz
-
Yes - that's exactly it. Can I just do an export of the campaign crawl? or is that found somewhere else? Forgive my ignorance...still new to the tool and just can't seem to find.
-
Are you looking at your crawl report? It is available in an Excel file which provides the exact information you desire.
-
Hey Ryan,
Thanks for the response.
It's not a huge site. I would say there's probably around 600 total pages. It definitely looks like the SEO crawler has crawled the entire site, but I can't seem to find a way to just get a one-shot view of all the pages it's accessed.
Any ideas on where this can be found?
Thanks again!
-
How many pages are on your site?
The SEOmoz crawler will do exactly what you want but there is a URL limit. You should also understand it will experience the same limitations of any crawler. It wont list pages which behind a user log-in nor pages which are blocked by robots.txt. I am pretty sure the SEOmoz crawler doesn't list secure pages either but last I heard they were working on including them.
If your site has any island pages they wont be discovered either.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I safely asume that links between subsites on a subdirectories based multisite will be treated as internal links within a single site by Google?
I am building a multisite network based in subdirectories (of the mainsite.com/site1 kind) where the main site is like a company site, and subsites are focused on brands or projects of that company. There will be links back and forth from the main site and the subsites, as if subsites were just categories or pages within the main site (they are hosted in subfolders of the main domain, after all). Now, Google's John Mueller has said: <<as far="" as="" their="" url="" structure="" is concerned,="" subdirectories="" are="" no="" different="" from="" pages="" and="" subpages="" on="" your="" main="" site.="" google="" will="" do="" its="" best="" to="" identify="" where="" sites="" separate="" using="" but="" the="" is="" same="" for="" a="" single="" site,="" you="" should="" assume="" that="" seo="" purposes,="" network="" be="" treated="" one="">></as> This sounds fine to me, except for the part "Google will do its best to identify where sites are separate", because then, if Google establishes that my multisite structure is actually a collection of different sites, links between subsites and mainsite would be considered backlinks between my own sites, which could be therefore considered a link wheel, that is, a kind of linking structure Google doesn't like. How can I make sure that Google understand my multisite as a unique site? P.S. - The reason I chose this multisite structure, instead of hosting brands in categories of the main site, is that if I use the subdirectories based multisite feature I will be able to map a TLD domain to any of my brands (subsites) whenever I'd choose to give that brand a more distinct profile, as if it really was a different website.
Web Design | | PabloCulebras0 -
Do changing theme would cause the the seo effects?
I want to ask that i want to change my website theme does this will have any side effects on my humidifier site Seo?
Web Design | | farhankhan110 -
Anyone using CloudFlare on multiple sites?
We are considering using CloudFlare as a CDN for a large group of sites. The fees are $5 to $200 depending on many factors. We tried the free trial on one site and were impressed with the results. I am wondering if any of you have any longer term experience with this and performance metrics, etc.
Web Design | | RobertFisher1 -
On site SEO opinions
Hi all, I have been testing different configurations for my on-site SEO for a while now and I think I am pretty much there. However it is always nice to know what other SEO's think about my keyword density and usage. My site is http://www.tomlondonmagic.com I am curious as to what you think regarding landing page content, whether you need lots or text or little text? I have just removed links in the text as I feel I want to keep as much juice on my landing page as possible. Thanks all!
Web Design | | TomLondon0 -
Can you use a base element and mod_rewrite to alleviate the need for absolute URLs?
This is a follow up question to Scott Parsons' question about using absolute versus relative URLs when linking internally. Andy King makes the statement that this can be done and that it saves additional space (which he claims then can improve page speed). Is this a true and accurate statement? Can using a base element and mod-rewrite alleviate the need for absolute URLs? I need to know before going off on a "change all of our relative URLs to absolutes" campaign. Thanks in advance! Dana
Web Design | | danatanseo0 -
Thoughts on how to speed up my site? (Other site ideas.)
I was wondering if someone had a few minutes to talk a look at our site and see how to speed it up. I've ran some of the speed tests and I get different results, so I'm not sure if my site would be considered "slow" or not. Also if there any other things that jump out at you (usability, side bar, things I should change / add / take away to make a users experience better, etc.) please let me know. Any feedback is good feedback to me. 🙂 I'm using WP Super Cache as well. Thanks!
Web Design | | NoahsDad1 -
E Commerce Product URL SEO
Hi, first question here. In response to Paddy's wonderful article and Rand's related article from 09 I've found myself asking this question. "www.home.co.uk/product" or "www.home.co.uk/category/product" As I'm currently reviewing the structure for so many of our e com sites the article comes at a great time. I'm going for home/category/product so I can optimise for a category, as category based searches are quite high and competitive. Yes some products can appear in 2 categories but I'm combating it with 301's or image based links. What are your thoughts.
Web Design | | PASSLtd0 -
New URL Strucutre
Hello all! I'm new to SeoMoz and would like to introduce myself and say thanks for all of the great content. This is definitely the premium website I have been looking for. My question is regarding a new URL structure that will be implemented with our website redesign. We are looking to make our URL's more user and SEO friendly. We will be using 301 redirects to reroute our existing structure when the site goes live. Background
Web Design | | kauffmantire
I work in a company that retails automotive tires. I would consider class (performance, off-road, highway, etc.) to be the most important category, but based on our analytics we receive much more traffic from brand terms. This is most likely because our customers are not educated on specific tire classes and generally do head term searches or search by brand / product name. Ultimately the will have to buy from a certain class based on driving habits. Link and URL Structure
(Parameters) The model page includes links to individual tire size specific pages. We are using the model page as our primary SEO page because the only difference from the 5 - 60 tire size specific pages per model will be the tire size itself. This means that all of the remaining content will be duplicated multiple times over. All links to the tire size specific page from the model page will have a rel="nofollow" attribute attached to it. All tire size specific pages will have a rel="canonical" tag pointing back to the model page Out of the following structures, which would you consider to be the best format: 1. http://www.company.com/category/brand/model
or
2. http://www.company.com/brand/category/model
or
3. http://www.company.com/tires/category/model We will append the specific tire-size to the URL on the item specific pages. Any other options or strategies will be greatly appreciated. Thanks in advance for your help!0