Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email [email protected] about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email [email protected] they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Over-Optimized Website
I'm looking for advice for what you would start with if you were working on a website that was extremely over-optimized for 1 keyword. So, for example, I'm going to pretend this client is a dog trainer in Toronto (I can't publicly post the URL). I've read places that having exact-match anchor text links to inner pages in the footer of the site can cause problems and removing it has resulted in big ranking jumps. I'm looking to see if there are other big items that you would tackle first if this was your client. Some examples of things the site has: There is a page for dog training under their Services menu. However, internal links on their site link "dog training" to both the homepage and to this service page. Is that going to cause issues? The anchor text for internal linking is almost always the exact same word - "Dog Training". There is a banner that goes across the top of the site that appears on every page that says "Dog Training Toronto". I'm guessing I should remove that. Would the same keyword being overly used on every page cause confusion? Almost every image on his site is saved in the format "Dog Training Toronto". I'm looking to see if anyone has general tips on where to start with a site that has been over-optimized for 1 keyword. He actually has a ton of good content on his blog that gets a ton of traffic (because it's actually useful) so it's not that his content sucks - it's just been overly structured and SEO'd to death. I found a few articles on this but other than the footer advice I didn't find too many case studies of others that have run into this issue and done a few steps that actually worked.
On-Page Optimization | | ImprezzioMarketing0 -
How is this page ranking?
Hi. A client of mine is being outranked by a competitor whose landing page does not include the keyword within their page content AT ALL. Nor does their URL. Nor do any image alts. And their page title features the keyword in the middle of it, not at the start. Their link profile is not great with directories and the like. They are not socially active.. I am confused! I thought content on a page absolutely had to include the keyword to get ranked for it. Here's the page: www.springsoft.ie, keyword is "water softeners" Any thoughts I would appreciate. Many thanks.Christoffa
On-Page Optimization | | Christoffa0 -
Understanding SEOMOZ and how to make improvements to my website
I want the same as everyone else the best position in Google. My campaign is set up as sub-domain. I am trying to get an understanding of what make the improvements to the following. What are the most important metrics and if I can work to improve them what makes those improvements. Currently I have a good position in Google for my most productive phrase but feel I need to make improvements to ensure I keep that position. I am adding regular unique content and looking for relavant quality links but it does not seem to be making much difference to the metrics below. Domain Authority 16 Domain MozRank 2.83 Domain MozTrust 0.86 External Followed Links 16 Total External Links 55 Total Links 668 Followed Linking Root Domains 8 Total Linking Root Domains 9 Linking C-Blocks 3 Subdomain MozRank 2.95 Subdomain MozTrust 1.73 External Followed Links 15 Total External Links 54 Total Links 667 Followed Linking Root Domains 8 Total Linking Root Domains 9 Which are the important metrics and can I improve my site to increase my Moz position therfore improving Google results
On-Page Optimization | | PhilSmith230 -
Home page keyword effecting internal page ranking
Hello, My client has a second keyword for the home page that is competitive. The home page is not being ranked for this keyword. Instead, an internal category page is ranking. This internal category page is more relevant than the home page - it shows the categories for the actual products that this term refers to. But everyone around us in Google's page results has far more backlinks than the internal page, and we're all heavily optimized for this term. My question is, is it safe to pull the second term off of the home page or is this internal page strong because it is somehow being strengthened by the home page optimization?
On-Page Optimization | | BobGW0 -
Is reported duplication on the pages or their canonical pages?
There are several sections getting flagged for duplication on one of our sites: http://mysite.com/section-1/?something=X&confirmed=true
On-Page Optimization | | Safelincs
http://mysite.com/section-2/?something=X&confirmed=true
http://mysite.com/section-3/?something=X&confirmed=true Each of the above are showing as having duplicates of the other sections. Indeed, these pages are exactly the same (it's just an SMS confirmation page you enter your code in), however, they all have canonical links back to the section (without the query string), i.e. section-1, section-2 and section-3 respectively. These three sections have unique content and aren't flagged up for duplications themselves, so my questions are: Are the pages with the query strings the duplicates, and if so why are the canonical links being ignored? or Are the canonical pages without the query strings the duplicates, and if so why don't they appear as URLs in their own right in the duplicate content report? I am guessing it's the former, but I can't figure out why it would ignore the canonical links. Any ideas? Thanks0 -
Framed Pages and Dynamic Pages
Has anyone else had experience with different CMS's for Ecommerce . Ones that create static pages and others that dynamically create pages. What differences have you seen with rankings on google with the two. Here are two examples of sites using static framed pages and one with a system that dynamically creates pages - http://www.gardeningexpress.co.uk/ - static frames and http://www.floraselect.co.uk - dynamically
On-Page Optimization | | onlinemediadirect0 -
Too many On-Page Links on a WP based Website
Hi, I've already browsed through various of the Q&As on the "too many On-Page links" issue, but I would really need some advice concerning a WP Site with a dropdown navigation. As outlined in the on-page report, every site has about 180 outgoing links, which pretty much is the number of site featured in the navigation. Even though the 100 link limit is somewhat outdated I'm still worried about the distribution of linkjuice from the starting page and how Google perceives the importance of the various pages. Would it make sense to adapt the structure of the navigation, so that the starting page only links to the 5 category pages and the category pages only link to the detail pages they contain? The site has good rankings for several pages and I assume that Google can tell that the large number of links is caused by the navigation. But with every page having appr. 180 links it may be difficult for Google to tell, which of those pages are the most important regarding internal link structure... Looking foward to your opinion and insights! Cheers, Chris
On-Page Optimization | | adwordize0 -
E-Commerce product pages that have multiple skus with unique pages.
Hey Guys, With the recent farm/panda update from google i'm at a cross roads as to how I should optimize product pages for a project i'm working on for a client. My client sells tires and one particular tire brand can have up to 15 models and each model can have up to 30 sizes. IE: 'Michelin Pilot Sport Cup' comes in 15 different sizes. Each size will have it's unique product page and description bringing me to my question. Should I use the same description on every size? I do plan on writting unique content for each tire model however i'm not sure if I should do it for every size. After all the tire model description is the same for every size, each size doesn't carry any unique characteristics that I can describe. Thanks in advance!
On-Page Optimization | | MikeDelaCruz770