It's that time of the month again! Mozscape's index is updated with fresh link information and Open Site Explorer, the Mozbar, PRO campaigns, and the Mozscape API all have new data.

This update featured some interesting work from our engineers, who've switched to new machines on AWS, which appear to be more stable and able to process our indices (unfortunately, they're about 4X the cost, but given our prior challenges with EC2, this is worth the price). We're still experimenting with our hosted cloud solution in Virginia, and we're hopeful that an index will be produced there by the end of the year. We'd originally hoped for earlier, but it's proving to be hard... Very hard :-)

Here are the metrics for this latest index:

•  55,852,954,589 (55 billion) URLs
•  1,017,642,509 (1 billion) Subdomains
•  114,764,072 (114 million) Root Domains
•  594,059,681,856 (594 billion) Links
• Followed vs. Nofollowed
  • 2.23% of all links found were nofollowed
  • 56.47% of nofollowed links are interna
  • 43.53% are external
• Rel Canonical - 12.98% of all pages now employ a rel=canonical tag
• The average page has 71 links on it
  •  64.15 internal links on average
  •  9.16 external links on average

And the following correlations with Google's US search results:

• Page Authority - 0.34
• Domain Authority - 0.24
• MozRank - 0.19
• Linking Root Domains - 0.24
• Total Links - 0.20
• External Links - 0.24

I've also got a histogram to show the crawl date and freshness of results in this index:

As you can see, most of the index was crawled in the beginning to middle of September, and there's ~1/3rd of the index from the latter half of August as well. Thus, this index will reflect links that occurred during that timeframe. As processing cycles get faster, the data becomes fresher, but we've got a lot of work to do to get there (thankfully, we also have a great team working on the issue).

As always, we'd love your feedback. Hope to hear from you in the comments, where our big data team will be reading and responding as usual!

p.s. Remember that if you're ever curious about when Mozscape is updating, you can check the calendar here. We also maintain a list of previous index updates with metrics here.