Web Diarists, Collaborative Filtering, and Scale-Free Networks

Even though it disparages Josh Marshall, we have to thank No Data Source for the new Hugh Hewitt piece on the Big Four web logs.

There was a time when web-based journalism was supposed to somehow revolutionize the delivery of news. The combination of low overhead and accessibility that web sites provide was supposed to wrest control of news from corporations and put it in the hands of the people. Now, presumably, anyone can publish their own broadsheet.

It’s unavoidable that readership is going to gravitate towards a small group of news providers — no one person can read everything. The decision of which news sources to read is influenced in large part by their visiblity and referrals from friends — it’s a textbook
scale-free network
, where things that are popular tend to stay popular, and the ignored stay ignored.

The result is Hewitt’s Big Four: Instapundit, Mickey Kaus, Andrew Sullivan and the Volokh Conspiracy. Together, these four news outlets exert an enormous amount of influence over the day’s agenda, reducing most publishers (like ourselves) to echoes and rehashings of thier posts. This is natural, of course — reputation and habit are an essential part of the intellectual economy. It’s also functionally identical to the “corporate media” problem: the agenda’s controlled by a handful.

It’s useful to look at how computer scientists deal with this “collaborative filtering” problem. After a time, ranking items by strict popularity becomes less useful. The homogenization of search results are going to prevent valuable but unknown items from being found. The simplest solution is to insert unpopular items, at random. This doesn’t interfere too much with the accuracy of the results, but does give a fighting chance to the underdogs.

For you, the news consumer, this means occasionally trying something new. Just visit WebLogs or another blog aggregate service, and see if you can’t find a new favorite. Unfortunately, scale-free networks tend to discourage this behavior. You need a large number of people accidentally picking up the same underdog at the same time in order to gather enough momentum to bring it to the top.

Epidemiology studies scale-free networks, too. Viruses get passed around by a core group, and infect populations in clusters. So, it seems, truth is a virus.