In October of 2014, Google launched "In the news", replacing their traditional news vertical results with a broader range of sources from across the web. Last week, Google's news results were shaken up again with the launch of "Top Stories", a card-style set of featured stories. Here's an example from a search for "John Glenn":

Even John Glenn's death somehow can't escape becoming a Trump story, but that's a topic for another time. What do we know about the shift to "Top Stories", and does this indicate a change in the way Google defines what's newsworthy? Let's start with the data...

Vital statistics

The following data was captured on Friday, December 9th across a tracking set of 10,000 keywords. These keywords cover a wide range of categories and types. Prior to the changes last week, "In the news" fluctuated on a weekly cycle (peaking mid-week), but occurred on somewhere between 10-15% of the keywords we track daily:

As of Friday, "In the news" had fallen to less than 2% of searches in the tracking set, and "Top Stories" spiked quickly to almost 13% (in the same range as "In the news" previously). None of the searches in our tracking set had both "In the news" and "Top Stories" on the same results page. It seems clear that "Top Stories" is replacing all news searches, and we can expect "In the news" to be completely phased out soon.

The new "Top Stories" UI has two distinct designs. The card-style design above accounted for 78% of the "Top Stories" results in our data set. The remaining 22% looked like this result for "flu symptoms":

Like the old "In the news" pack, the vertical "Top Stories" list can have from one to three stories. The horizontal version has three stories in every example in our data set (1,011 total).

The Newsmakers

Who's making the news that makes it into "Top Stories"? Across our tracking data set, we recorded 3,605 URLs appearing in "Top Stories" (some were duplicates, appearing across more than one search). Those 3,605 stories came from 1,319 different domains, suggesting that "Top Stories" is still sampling a very broad set of sources. These were the top 10:

1. Forbes.com (2.1%)
2. NYTimes.com (2.0%)
3. USAToday.com (1.7%)
4. FoxNews.com (1.2%)
5. Bankrate.com (1.0%)
6. Reuters.com (1.0%)
7. WSJ.com (1.0%)
8. CBSNews.com (0.9%)
9. CNBC.com (0.9%)
10. WashingtonPost.com (0.9%)

The top 10 sources accounted for almost 13% of all stories, and the top 50 accounted for just under 25%. The top 10 generally represented reputable news sources, although you might not think of Bankrate.com as a news source. Bankrate is appearing on commercial searches, such as this one for "buy cars":

In the context of that particular search, these are fairly reputable sources, but the search itself isn't one we would usually think of as newsworthy. Like "In the news" before it, Google seems to be casting a wide net with "Top Stories".

The News-fakers

Given the recent interest in fake news stories, some people have speculated that "Top Stories" is Google's attempt to address dubious news sources. Google has been fairly quiet so far about the motivation behind "Top Stories", but my sources suggest that this is primarily a design/UI change. This argument is supported by the fact that "Top Stories" rolled out with a broader redesign, including a new header and updated designs for image results and Twitter results. Additionally, "Top Stories" has been in testing on mobile search for a few months now, prior to the desktop roll-out

Our data also suggests that the number of sources involved in "Top Stories" is still very broad. Is there any evidence of some of the more controversial news sources of 2016? A quick check shows a handful of "Top Stories" results for Breitbart.com, with searches ranging from "second amendement" to "Kellogg's":

This is a bit of a grey area – while the Kellogg's boycott story is certainly newsworthy, and Breitbart's announcement is the source of that boycott, many people would consider them far too biased to quote as a source on this particular topic. Their inclusion suggests that Google is tapping a very wide range of sources from web results, without necessarily vetting the content of those sources.

Politics aside, other clearly fake sources are popping up in "Top Stories". Look carefully at this set of three results on a search for "violin":

While I'm personally a fan of The Onion, I think it's safe to say that no one should consider the article in the third card to be actual news. The second article, while certainly legitimate, is clearly an opinion piece. Our data set also included a handful of articles from Snopes.com. Here's one on a search for "lyme disease":

I'm sure we're all relieved to know that our Christmas Trees are (probably) not infested with disease-carrying ticks, but while Snopes has become a credible source for debunking bad information, it's certainly not a traditional news source in most people's eyes.

What is news?

Google's job isn't easy. News is no longer something delivered to us in a 30-minute nightly television program, recapping the same world of hand-picked information for everyone. News is contextual and driven by the information we seek. For example, almost no one would consider Nintendo.com to be a news source, yet take a look at the following set of "Top Stories" results:

These results appeared on a search for "3DS games" – in that context, a recent release on Nintendo's site is both timely and, in a broad sense, newsworthy. We wouldn't expect a link to Nintendo when searching for news about the conflict in Syria, but in this context they're a reputable source.

While "Top Stories" may be primarily a design overhaul for now, I do think that the conscious removal of the word "news" signals Google's intent. They have to be able to deliver stories in any context, even when traditional, reputable news sources aren't available (or, at least, when the reputation of those sources is unknown).

If Google sets the reputation threshold too high or restricts "Top Stories" to hand-picked sources, then they'll only be able to deliver recent news results to a very small set of searches. If they set it too low, then we'll be inundated with fake news. We're still a long way from teaching a machine to fact-check, and Google has a tough road ahead.