Website pages missing from seomoz crawl
-
Hi!
I just added a website and the crawling result output has only 42 pages but my website has about 75 pages. What am i missing?
Thanks!
-
Is there any chance you could email me your sitemap as produced by wordpress? info[at]pathfindermedia[dot]co[dot]uk I'll take a closer look at whats being excluded.
-
I'll do that Keri. Thanks!
-
I don't think so:
User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Goole Bot User-agent: Googlebot Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ # Google Image User-agent: Googlebot-Image Disallow: Allow: /*
Regards.
-
Another possibility could be your robots.txt file, is it blocking some directories?
-
Hi! It's probably best to email [email protected] about this. You can give them your full URL and they can help figure out why Roger isn't crawling everything. Thanks!
-
Anyone?
-
Hi!
One thing i figured out is that the crawling on both seomoz and xml-sitemaps.com returm the same 42 pages.Here's my website homepage URL - http://bit.ly/TGjpVx
And a couple of missing pages from 36 at total - http://bit.ly/WM3Rwe and http://bit.ly/VpHJ9H.
Regards,
OV -
Hi!
My website has a xml sitemap, generated by Google XML sitemap Wordpress plugin, with all the 75 pages.The crawler http://www.xml-sitemaps.com/ also outputs just 42 pages. I think it has someting to do with the blogs being archived (?).
I need to solve this and don't know how?!
Thanks for your help do.
Regards.
-
Actually, the SEOmoz crawler should be crawling all of the pages -- it's OSE that doesn't crawl everything, but the crawler from your campaign should show all that it could find. If you email [email protected] they'd be happy to help you figure it out, or if you want to share your URL here along with some pages that are missing, the Q&A people could help diagnose things too.
-
Hi Ovieira,
This is not necessarily an indication that there are pages that are hidden from crawlers and the missing pages could simply be low priority for the moment. Or could have been created after the initial crawl had taken place.
The best way to check is to run a crawler like http://www.xml-sitemaps.com/ and that will give you a better idea. If the sitemap generates a complement of your pages then it's probably just a case of waiting until the next Moz crawl.
Mulith
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Should I combine pages?
Hi, Im not sure of the correct route to take here... We are a training provider and I manage the website. The main course offered is the transport manager CPC. Currently, I have a "catch all" landing page which links to each different course option: Landing page > Classroom Online Self study Distance learning The main keyword revolves around "transport manager cpc" I want searchers to land on the online page is they search "online transport manager CPC" for example but I think its confusing Google. I'm wondering if I should de-index the store pages (although some perform very well) and increase the content on the main landing page to rank for every related keyword on that page. Initially, I wanted to devalue the landing page in favor of the store pages but I'm unsure if that's the right way to go. I've stripped out the bulk of the keywords and content and shifted it to each individual page. but as above, Im now unsure if that's the right route to take. Any help would be greatly appreciated 👍 Thanks
On-Page Optimization | | dunbavand
Rich0 -
Homepage On-page Optimization
How do you all handle homepage optimization, if you (or a client) offers a variety of services? Our homepage has the strongest link profile of any of our pages, but it lists all the areas of law we cover. Therefore, it has too many keywords and none really rank well. Should we just pick our most profitable areas and optimize for that? www.kempruge.com in case anyone would benefit from looking at the actual page. Thanks, Ruben
On-Page Optimization | | KempRugeLawGroup0 -
How to treat pages that are removed?
I have a website that need be very up-to-date, I mean, pages can be published just for 30 days, after that it should be unpublished. Everyday more than 300 pages is "removed", For theses pages I am returning http code "410" (Gone), also I remove from the sitemap. Now, I am checking Google WebMasterTools and I am getting thousands of pages not found. So... My questions Does it have SEO impact? How is the best approach to treat it?
On-Page Optimization | | thobryan0 -
Google Indexed = 35, 445 pages, Bing Indexed = 243 pages... Why?
Dear MozSquad, Can anyone check our site and let me know if there's anything super apparent that would cause Bing to treat us like a bum on the street? I recently made some structural changes which really helped with Google, but Bing didn't even budge. It's a lot harder to keep up with all the SEO initiatives I have in mind with it being a small start-up where I'm responsible for planning the entire Internet Marketing campaign, giving constant input on UX and site design, etc on top of 900 other things, so I figured it'd be a good time to use The Moz to help a brother out. Ideas? Domain: homeandgardendesignideas.com (yeah, I know it's a little long =P)
On-Page Optimization | | zDucketz0 -
Why is SEOMOZ Crawl Diagnostics not in sync with Webmaster Tools
Currently, my Website, according to the Crawl Diagnostics Summary, has 401 'Duplicate Page Title Errors'. But in Google Webmaster Tools, under Óptimization on the Left hand Side Toolbar, if you look up HTML Improvements, there are only 4 'Duplicate Title Tags'. I have two questions re this: A) Do 'Duplicate Page Title Errors' and 'Duplicate Title Tags' have the same meaning' ? , and B) why are there 401 errors located by the former, and just 4 by the latter?
On-Page Optimization | | ABCPS0 -
Website title question
Say you have a website url of a rather competitive keyword phrase, would it be beneficial for me to go ahead and name my site title the same as the url? And also should my site title go through every page, or should i consider having slight variations throughout the pages? for example: page title | site title or page title| slight varation of title on sub page? **edit - to further expand on the question a bit also, if my google places has the company name on _there - would it be effective to go ahead and use the company name in my site title? _ _Also if i have the main keyword in the breadcrumb as the home, does that effect my SEO credibility if it shows up on all the pages? _
On-Page Optimization | | tgr0ss0 -
On Page SEO Tool
Hello - I'm looking for one tool that does the following and was wondering if anyone knew of such a tool? In a perfect world I would like to enter in one domain name and have a report generated that shows All Internal links, link titles, and anchor text All internal broken links / redirects All images, image size and image alt, if the image alt is missing. I'd love to be about to export these reports to excel and quickly run my on page optimization. The goal is to produce a checklist for a developer to execute quickly. Thanks for your help Gabe
On-Page Optimization | | Gabe0 -
Landing page too long
In my first seomoz test I have many tilte pages for products that are over 70 charachers long. The part #'s are long like 10-782-10-10-10-PPxxxx etc. All these part #'s are not my key products and I could delete or truncate. My question is if the part numbers are not that important, is it OK to leave them as is or is ranking being damaged because they exist?
On-Page Optimization | | Wales0