The Big Data Team at Moz has had one of the hardest periods to date producing an index (and we've had some nightmarish doozies in the past). In June, after our funding from Foundry & Ignition, we started running 4 simultaneous indices on Amazon while also starting to set up our new hybrid (our processing and Amazon storage) cloud data center in Virginia. This tactic mostly worked until August, when we had 4 indices collapse on us due to a high rate of Amazon EC2 disk failures. Naturally, this makes us want to move off the cloud more quickly, and it also meant our bill was higher than ever - nearly $700,000.

Here are the metrics for this latest index:

• 64,023,562,478 (64 billion) URLs
• 1,282,691,523 (1.2 billion) Subdomains
• 148,634,588 (148 million) Root Domains
• 651,894,828,133 (651 billion) Links
• Followed vs. Nofollowed 2.28% of all links found were nofollowed
  • 55.53% of nofollowed links are internal
  • 44.47% are external
• Rel Canonical - 13.74% of all pages now employ a rel=canonical tag
• The average page has 71 links on it
  • 60.74 internal links on average
  • 10.87 external links on average

And the following correlations with Google's US search results:

• Page Authority - 0.34
• Domain Authority - 0.24
• MozRank - 0.20
• Linking Root Domains - 0.24
• Total Links - 0.20
• External Links - 0.24