Welcome to the Q&A Forum

Remember the technique Rand posted for counting the unique indexed landing pages on a site?

http://moz.com/blog/indexation-for-seo-real-numbers-in-5-easy-steps

I'm using Google Analytics to compare this result for the "social" channel vs. the "organic search" channel. The idea is to see whether social or search delivers people to more unique landing pages.

I would expect organic search to yield a significantly higher count, due to long tail searches for relatively obscure terms. We have an archive of over 500k pages that Google could be indexing, even if only a fraction of a percent of those might be getting social mentions. But that's not what the numbers show -- I'm seeing roughly 10% higher numbers for the Social channel than for Organic Search. Again, I'm counting unique landing pages here, not the total pageviews or visits.

If anyone else here is monitoring these metrics, please chime in with your results. What's the baseline expectation for a large website with hundreds of thousands of pages of issue-centric user-generated content? Should search traffic provide 2x more landing page exposures than social? 5x? 10x? or 0.9x, as in my case?

In short, I believe these numbers are pointing to an indexing/ranking problem following a recent site redesign.

Our content is occasionally featured in Google News. We recently have had two episodes where this happened, but (a) nearly all the referring URLs were blank, and (b) our backend logs show 3-4x more requests for the article in question than Omniture does.

In other words, hundreds of thousands of visitors requested a URL from our site (as proven by the traffic logs), but don't seem to have come from Google News (because HTTP_REFERER was blank), and didn't execute the onpage javascript tag to notify Omniture of the pageview.

Perhaps this has nothing to do with Google News, but it is too strong a coincidence that the two times we were on there recently, the same thing happened: big backend traffic spike that is not seen by Omniture.

It is as if Google News causes browsers to pre-fetch our article without executing the javascript on the page. And without sending a referring URL.

Has anyone else seen anything like this before?

Stats from the recent episode:
 - 835,000 HTTP requests for the article URL (logged by our servers)

- these requests came from 280,000 distinct IP addresses (70% US)

- the #1 referring URL is blank. This accounts for 99.4% of requests. Which, in itself, is hard to believe. These people had to come from somewhere. I believe browsers don't pass HTTP_REFERER when you click from an SSL page to a non-SSL page, but I think Google News doesn't bounce users to SSL by default.That said, we do see other content pages with 70-90% blank referring URLs. Rarely 99+% though.

The answer depends on whether there exist any links in the search engines' link graphs that point to those old product pages.

If you have offsite links to old pages, your 301s need to stay in place. Otherwise, whatever link juice is passed from those 3rd-party sites will stop being credited to your new product pages. If there are a significant number of such links, your new pages could drop in the rankings.

(To find links to those old product pages, use OSE.)

SEOs know that a rapid uptick in link acquisition speed can trigger a red flag within google.

Is the same true of a FB Like campaign?

I'm not up to speed on whether Google is believed to use Facebook Likes in its ranking algo. Should we treat a FB Like acquisition campaign like we do linkbuilding -- namely, throttle it so that there isn't a huge spike in growth rate for a short period?

Surely FB might have internal algos that devalue 'astroturf' likes, but the core concern of this client is whether SEARCH RANKINGS might suffer if the FB Likes grow too quickly. My gut says no, don't be concerned. What do you think?

Ryan - not half the site's traffic, but half the site's search traffic. And even that is an exaggeration. Webspam search traffic accounts for 28% of overall search traffic.

EGOL - I would say no to the question of robot visitors, because on the instances we checked -- in which spammers used a bit.ly URL for their outbound link -- we were able to measure an astounding 47% clickthrough rate from our site to the spam destination. I would not expect bots to click through.

Also, we use nofollow on all outbound links in user-generated content. I guess that is not a guarantee that we would not be penalized fro hosting a linkfarm, but shouldn't it  be?

If it were up to me, I'd wipe out the webspam entirely. But it's not an easy sell. This content delivers ~750,000 pageviews, ~150k ad views, and probably 100k unique visitors per month, plus the small risk that one day G might penalize us for it. It's not pills, porn, gambling, mortgages, and all the links are nofollowed. The people making this decision don't see a smoking gun.

Let's say your user-generated content strategy is wildly successful, in a slightly twisted sense: webspammers fill it with online streaming sports teasers and the promise of "Weeds season 7 episode 11." As a result of hard SEO work done to build the profile of the domain, these webspam pages seem to rank well in Google, and deliver nearly 750k pageviews, and many many unique visitors, to the site every month.

The ad-sales team loves the traffic boost. Overall traffic, uniques, and search numbers look rosy.

What do you do?

a) let it ride

b) throw away roughly half your search traffic overnight by deleting all the spam and tightening the controls to prevent spammers from continuing to abuse the site

There are middle-ground solutions, like using NOINDEX more liberally on UGC pages, but the end result is the same as option (b) even if it takes longer to get there.

The SEOmoz CDN appears to have a "pull zone" that is set to the root of the domain, such that any static file can be addressed from either subdomain:

http://www.seomoz.org/q/moz_nav_assets/images/logo.png

http://cdn.seomoz.org/q/moz_nav_assets/images/logo.png

The risk of this configuration is that web pages (not just images/CSS/JS) also get cached and served by the CDN. I won't put the URL here for fear of Google indexing it, but if you replace the 'www' in the URL below with 'cdn', you'll see a cached copy of the original:

http://www.seomoz.org/ugc/the-greatest-attribution-ever-graphed

The worst-case scenario is that the homepage gets indexed. But this doesn't happen here:

http://cdn.seomoz.org/

That URL issues a 301 redirect back to the canonical www subdomain. As it should.

Here's my question: how was that done?

Because maxcdn.com can't do it. If you set a "pull zone" to your entire domain, they'll cache your homepage and everything else. googlebot has a field day with that; it will reindex your entire site off the CDN.

Maybe the SEOmoz CDN provider (CloudFront) allows specific URLs to be blocked? Or do you detect the CloudFront IPs and serve them a 301 (which they'd proxy out to anyone requesting cdn.seomoz.org)?

One solution is to create a pull zone that points to a folder, like example.com/images... but this doesn't help a complex site that has cacheable content in multiple places (do you Wordpress users really store ALL your static content under /wp-content/ ?).

Or, as suggested above, dynamically detect requests from the CDN's proxy servers, and give them a 301 for any HTML-page request. This gets complex quickly, and is both prone to breakage and very difficult to regression-test.

Properly retrofitting a complex site to use a CDN, without creating a half-dozen new CDN subdomains, does not appear to be easy.

Regarding the question of testing a site on various mobile platforms, see this article about mobile emulators:

http://mobiforge.com/testing/story/a-guide-mobile-emulators