Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email [email protected] about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email [email protected] they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Dividing a website into two websites
I have a website which talks about Yoga and GMAT classes (2,000 pages). It has blog posts, Faqs, discussions, videos etc.I am planning to fork GMAT content (1,000) pages into a niche website to rank better. Is this a good idea or bad idea? Will Google think niche new Yoga site as duplicate content was there on Dance site in the beginning. I am doing this with assumption that niche website will have higher authority so that I can beat the competitors. What is the best approach to beat your competitors? I am primarily concerned of what happens if I fork half of the website into a new one. Will Google treat new website as a duplicate content website because, this content WAS present in old website and Google will have archives of that. Please help
On-Page Optimization | | Avinash_12340 -
Why doe SEOmoz include my categories and subcategories when counting on page links?
Hi everyone, I am new to SEOMOZ, but it looks like really good program. I ran a report last week, and SEOMOZ says that out of 10001 pages, I hve a total of 9995 warnings. When I take a look at the breakdown, it says I have too many on page links. I removed the links from the bottom of each page, but my on page links are still over 100. Does this mean that I will have to get rid of categories and sub-categories in the navigation bar? Thanks, Don
On-Page Optimization | | ge01734000 -
Third party pages
Suppose you are using a third party tool such as an affiliate program. Typically, all the files are organized under one subdirectory. In addition, you may have little or no ability to modify any of the files in terms of SEO. Would you recommend hiding the entire subdirectory with a noindex? Best,
On-Page Optimization | | ChristopherGlaeser
Christopher0 -
Google is indexing spam pages from my site. What is the most effective way to get ride of the search results? Pages are deleted now but should I do something more?
A long time ago I created a forum (Invision Power Board) and it got full of spam. Massive amounts! /forum/ I've now deleted the forum but the spam pages are still indexed on Google. Can I do something else to hurry up the process to get ride of them?
On-Page Optimization | | ocarlsson0 -
Too many links on a page?
On my blog posts, I have links to all the categories and months, dating back 5-6 years. This make the number of links on each blog page well over 100, which I understand might decrease the value of each page. Is there a problem with having more than 100 links on a page?
On-Page Optimization | | rdreich492 -
Blog page outranks static page for KW -- why?
Blog page ranks 10 in Google, while the static page is on page 7. What makes it more interesting is that the blog page scores an "F" with the Term Target tool while the static page scores an "A". Static page has more inbound links and a mR/mT of 3.89/ 4.54 vs. 3.71/ 4.14 for the blog page. Any ideas on how to approach this one?
On-Page Optimization | | 540SEO0 -
Organic Landing Pages...
For one of our sites (fastcubes.com) I noticed our landing pages were ranking and getting a few organic visits. Considering they were made specifically for PPC, I thought maybe we should create landing pages that would not be present in the navigation of the site but for the purpose of optimizing for keyword variations. For example work station cubicles vs office workstation. We have a page optimized for office workstation but having another optimized for work station cubicles is redundant. Would it be a good idea to create this as a page that is not present in the navigation for the sole person of hopefully being ranked and getting traffic for that specific keyword? Thank you in advance for your help!!
On-Page Optimization | | DevonIntl0 -
How SEOMOZ manage to have same pagerank on all pages?
I was just wondering how SEOMOZ manage to have same pagerank on all pages? Even on profile pages.
On-Page Optimization | | DiamondJewelryEmpire0