Solving feed overload

Ilya Grigorik has some thoughts on how to solve the information overload that often comes with feed reading. If you subscribe to hundreds of feeds, how do you find the interesting stuff without reading everything? How do you separate the signal from the noise.

Keyword filtering is one common approach. But there’s some significant problems.

I use keyword monitoring to let me know on new product announcements and to track public mentions of certain people or products. It does work in that instance. However, the act of me specifying the keyword presupposes a priori knowledge of the topic I would like to track. The reason why I cast my net so wide across so many sources is exactly because I don’t know what I’m looking for - I just want to make sure that once something significant happens, I’ll know about it! Not only does keyword filtering fail here, but the act of choosing the keywords is extremely hard in itself.

To some extent, feed readers appear to try and justify their existence (and boost their usage) by recommending feeds to watch. (Note that we’re sometimes guilty of this at Feed Crier, too) Suggesting additional feeds to read based on what you’re already reading is only contributing to the problem. Ilya rightly takes we feed reader developers to task, asking us why our suggestions are so self-serving, when we should be helping our users.

One approach that Ilya rejects is allowing our friends to decide what I read and what I see. He suggests that doing so places a burden on them and only works if we’re using the same system anyway. I’ve found that this is not the case, but only when looked at from a slightly different angle.

Before we descended into the echo chamber, blogs were a log of interesting places on the web. That’s the root of the term even: web log. The early blogs were human filters for the vast web. People chronicled their daily journey through the intarwebs, leaving breadcrumbs for others to follow. Almost three years ago I suggested that enterprise blogging could perform the same function for intranets by allowing employee blogs to act as human filters for the most important things.

Human filtering can work. I started subscribing to Roland Tanglao’s blog several years ago because he was an effective filter for high-output bloggers like Dave Winer and Robert Scoble. He separated the signal from the noise—if something interesting was said on a vast number of blogs, I could count on Roland to reblog it for me. His blog as since morphed into his personal interests, but there are other bloggers that do the same things.

Ilya suggests two approaches, building smarter feed readers and an interestingness measure that comes from the community.

Shifting the burden on the RSS reader could yield some improvements. Clustering, correlation, prior reading patterns can all be taken into account when information is sifted through the filters.

He recognizes the problem with a smart reader—that it takes a fair amount of training to start doing it’s job. A reader needs to watch your habits and learn what you find interesting, something that takes time. My friends at Touchstone are working on solving the problem in part by passively watching what you do on your computer and using that information for training. This passive approach can ease the training burden. Training in this case doesn’t require you to do anything different than what you are already doing. Their attention engine looks at what you’re already paying attention to and helps you find other things that you’d find interesting.

A community interestingness measure is what memetrackers are working on. Tailrank, Techmeme, and others (disclosure: I’m an advisor to Tailrank) are using the linking patterns of blogs to determine what the community at large finds interesting. Tailrank takes this a step farther by helping you find the interesting things in the feeds you’re already reading. Their My Tail service allows you to tell them what blogs you’re reading and get a customized page and feed of the most interesting items in those blogs. And the clustering behavior of memetrackers can help readers find related topics and other points of view.

I have a nagging concern, however, that these systems designed to reduce information overload could reduce serendipitous findings as well. If all you see is things that you already find interesting, how do you expand your horizons? By reading a vast and eclectic collection of feeds, I’ve been exposed to new and interesting ideas—ideas that I wouldn’t have even known I’d have liked. Will attempts to reduce information overload contribute to creating a monoculture?