Getting your site totally indexed by SEOMOZ
-
Hi guys!
Ijust started using SEOMOZ software and wondered how it could be that my site has over 10.000 pages but in the Pro Dashboard it only indexed about 1500 of them. I've been waiting a few weeks now but the number has been stable ever since. Is there a way to get the whole site indexed by SEOMoz software?
Thanks for your answers!
-
Thanks Abe!
As soon as I can analyse the whole site, including all the links, i will get back at you.
I think you got a good point about the links. I don't think SEOmoz can do anything about that
Greets,
-
Hello,
I'm sorry that some of your pages didn't end up getting crawled. There's a lot of reasons why this could have happened, some of which have to do with our crawler, and others that have to do with a site. Basically, the crawler finds pages by crawling all of the links on the homepage/front page and then crawling all of the pages those links took him to, and so forth, until all of the pages we can find under that directory are crawled. Usually, if a page is linked to from the homepage, it should end up getting crawled. If it doesn't, it's either a sign that those pages aren't as accessible as they could be to search engines, or that there is an issue with our crawler. Some of the time, it's a problem with Javascript - our crawler can't handle Javascript very well right now. Other times, content was created a little bit after we crawled but before we finished processing and putting up the results. That content will be captured in the next crawl usually. It can also be due to meta tags or robots.txt telling rogerbot not to crawl.
On the whole, I'm not sure why some of your pages wouldn't be showing up in the crawl. Can you let me know some examples of pages that should be in the crawl but didn't end up in your campaign? If you could also outline the links going to those pages, I can send all that information over to our engineering team and we can try to diagnose the issue. Thanks a ton!Talk to you soon,
Abe Schmidt-
SEOmoz Helpster
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can check my site
can you please tell me what should I do with this site I trying a lot to solve its issues. the link is here https://alazharquranteaching.com/
Moz Pro | | saharali150 -
Large site with content silo's - best practice for deep indexing silo content
Thanks in advance for any advice/links/discussion. This honestly might be a scenario where we need to do some A/B testing. We have a massive (5 Million) content silo that is the basis for our long tail search strategy. Organic search traffic hits our individual "product" pages and we've divided our silo with a parent category & then secondarily with a field (so we can cross link to other content silo's using the same parent/field categorizations). We don't anticipate, nor expect to have top level category pages receive organic traffic - most people are searching for the individual/specific product (long tail). We're not trying to rank or get traffic for searches of all products in "category X" and others are competing and spending a lot in that area (head). The intent/purpose of the site structure/taxonomy is to more easily enable bots/crawlers to get deeper into our content silos. We've built the page for humans, but included link structure/taxonomy to assist crawlers. So here's my question on best practices. How to handle categories with 1,000+ pages/pagination. With our most popular product categories, there might be 100,000's products in one category. My top level hub page for a category looks like www.mysite/categoryA and the page build is showing 50 products and then pagination from 1-1000+. Currently we're using rel=next for pagination and for pages like www.mysite/categoryA?page=6 we make it reference itself as canonical (not the first/top page www.mysite/categoryA). Our goal is deep crawl/indexation of our silo. I use ScreamingFrog and SEOMoz campaign crawl to sample (site takes a week+ to fully crawl) and with each of these tools it "looks" like crawlers have gotten a bit "bogged down" with large categories with tons of pagination. For example rather than crawl multiple categories or fields to get to multiple product pages, some bots will hit all 1,000 (rel=next) pages of a single category. I don't want to waste crawl budget going through 1,000 pages of a single category, versus discovering/crawling more categories. I can't seem to find a consensus as to how to approach the issue. I can't have a page that lists "all" - there's just too much, so we're going to need pagination. I'm not worried about category pagination pages cannibalizing traffic as I don't expect any (should I make pages 2-1,000) noindex and canonically reference the main/first page in the category?). Should I worry about crawlers going deep in pagination among 1 category versus getting to more top level categories? Thanks!
Moz Pro | | DrewProZ1 -
Website not being indexed by Google - does seomoz have a index checker?
i have been to a couple of index checking websites that are saying my website is not being indexed, my developer and seo team have been making a lot of changes recently and I think it is because of that. Also my first byte grade is D. Can I check here on seomoz if my site is being indexed by Google here in the UK?
Moz Pro | | Malx0 -
How do I see all links to my site in Web Explore?
It appears that the Web Explore looks for exact matches to a page. I'd like to see all matches to any page. Will regular expressions work?
Moz Pro | | Leverage_Marketing0 -
SEOMoz Private Message Settings
Hi all, Is there a way to be notified (to my personal email address) when I get sent a private message? Thanks,
Moz Pro | | Unity
Davinia0 -
SEOMoz Q&A having some issues?
Apologies if this has been asked. I was browsing Q&A and got a Ruby error page, reloaded and everything was fine. However a bunch of the topics are kind of messed up, and it looks like a chunk of them are missing now. For example my 'questions I've answered' section of My Q&A only has the newest question and the oldest question, and when I sort the SEOMoz Tools category by newest I get one recent post and then the next one is from November of last year. You guys having database problems?
Moz Pro | | icecarats0 -
Why does it take so long for SeoMoz to update data?
I changed the Anchor Text of 4 40/100 MozRank sites 2 months ago, yet SeoMoz still shows the old Anchor text in the reports. Why is this taking so long? I also notice my inbound domains hasn't increased nor has my MozRank in 3-4 weeks. What's the turnaround?
Moz Pro | | sanchez19600 -
Canonical issue in open site explorer
When I look at my back links in OSE, I see two landing pages on my site that are really the same page. www.mysite.com/ and www.mysite.com/(affiliate code here) These show different inbound link characteristics and page authority. The page in question has a rel=canonical tag. Am I doing something wrong?
Moz Pro | | EugeneF0