Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email [email protected] about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email [email protected] they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I remove a high traffic page on my website?
For the last few years, a particular blog post on my site has gotten 3 times as much traffic than any other page, even the home page; however, the topic of the post is only moderately related to the website topic and I'm wondering if all that unrelated traffic is negatively effecting SEO for our primary keywords. Here's an example.... Site topic: Yoga retreats in Costa Rica (we want to attract people who are interested in booking a yoga retreat) Blog Topic: How to extend your visa in Costa Rica (it's related only because it's about Costa Rica and travel, and may help our visitors stay longer) Other Notes: In 4 years, visitors to that blog post have never converted. Blog post bounce rate is 56%, significantly higher than almost any other page Lots of comments on the blog post so visitors to it are engaged and find it very useful To get an accurate reading of interested visitors to the site, i always have to filter entrance visits to this post in my analytics because these users are not an accurate representation of the visitors we're trying to draw. My question: Because I get so much traffic from the blog post, which is about the visa renewal process, will Google consider the website less about yoga and more about visas? If so, will it make it more difficult to rank well for yoga in Costa Rica? Does Google say to itself, "Hey, this website can't be an authority about both yoga and visas in Costa Rica so we're going to consider it a visa site because of all the visits and engagement it gets for that topic." So should I remove the post or just leave it alone? It offers a lot of people valuable information so I would never delete it entirely, but would redirect it somewhere else. Thanks!
On-Page Optimization | | Cabaretti0 -
Page Hierarchy Question
I understand the basic concept of page hierarchy, i.e. parent and child pages. My question is: Should the home page be the parent of all 2nd-level pages? Can/should there only be one top-level page, the home page? In other words, is this: site.com/homesite.com/home/products site.com/home/products/widgetsite.com/home/aboutsite.com/home/contactbetter than this:site.com/homesite.com/products site.com/products/widgetsite.com/aboutsite.com/contactThanks for your opinion!
On-Page Optimization | | BillWoods0 -
Content for the Home Page
Hi All, I have a Videos website which contains Videos of all types + Family safe type... The home page has sections and Videos listed. Now for SEO purpose i need to have content? this is what i read in most places. What is the kind of content i can place on a Videos website Home page? I can write about a Movie or actor but that content on Home page would that be of any use? We have a About us page etc to know who we are.. Any ideas please..
On-Page Optimization | | Nettv0 -
Different pages for OS's vs 1 Page with Dynamic Content (user agent), what's the right approach?
We are creating a new homepage and the product are at different stages of development for different OS's. The value prop/messaging/some target keywords will be different for the various OS's for that reason. Question is, for SEO reasons, is it better to separate them into different pages or use 1 page and flip different content in based on the user agent?
On-Page Optimization | | JoeLin0 -
Dupelicate content home page and custom page question
I am working on a website that got hit by the penguin update. Didn't get hit terribly bad, but dropped from number one to number 9. As I'm going through the pages, the theme and content is a mess. To give an example, say the site is about custom colored marbles. The main page content covers custom colored marbles, custom promotional marbles, custom glass marbles, etc. Custom colored marbles is mentioned and covered on all pages, which I am going back and trying to make each page theme specific. There is also a custom page, so I am at a cross roads on how best to employ the focus of the custom page and the home page. I am thinking the home page should emphasize colored marbles, and the custom page should emphasize custom colored marbles. My fear is that making such a drastic change will bounce the site completely off front page and that it will take time for the custom page to come up in rankings. AS it stands now I am confused as to how it even ranks on first page as there's two pages with custom colored marbles emphasis. Id like to clean this up as much as possible so there are no big hits with future google updates, but I don't want the site to drop off either as that would be hard to explain to the owner. Yeah, we are cleaning up your site and making it google compliant and in so doing you no longer rank on first page. That won't put food on the table. Thanks for any advise on this.
On-Page Optimization | | anthonytjm0 -
Duplicate page
Just getting started and had a question regarding one of the reports. It is telling me that I have duplicate pages but I'm not sure how to resolve that.
On-Page Optimization | | KeylimeSocial0 -
Seomoz tabbed search
Hi, this question might not be SEO specific but i was wondering how seomoz implemented a direct search from the chrome omnibox (you write www.seomoz.org and the press TAB)? BR Carl
On-Page Optimization | | careeron0