Topix Blog

August 27, 2004

The Daily Internet

by skrenta at 8:47 AM

Jeremy Zawodny expands on a review of feed search engines: Why Hasn't Anyone Figured Out How To Do Feed Searches?

The kind of searches I regularly do on Feedster and Technorati just aren't available on Google. No amount of fiddling with the advanced search options, rooting around on their labs site, or searching for obscure options will scan incremental new material from half an hour ago. Yahoo search seems to have surpassed Google with some advanced features, but they don't have an effective reverse chronological "sort by date" either.

Google has been experimenting with crawling rss feeds for some time, but we haven't seen any blog or feed search from them yet. Google News and Freshbot suggest they clearly have the technology to roll out a best-of-breed version of feed search. And they certainly don't hold back when it comes to trying out live product experiments. Scott Rafer of Feedster disagrees, and argues that getting feed search right is harder than it looks. He may have a point; clicking "sort by date" in Google News often sucker-punches the relevance. But still it seems odd, that with the biggies supposedly in cutthroat competition for search, that they've left the field for Feedster as the best resource for this class of searches. Why?

Feeds & Blogs: Fad or something big?

In the 90's, Usenet was the largest distributed message board system on the net, and I depended on it and DejaNews (which provided Usenet search) to learn about new technology, troubleshoot problems, and research buying decisions. Folks on Usenet had something to say about everything -- cars, restaurants, apartments, houses, TVs, brands of computers, etc. -- and it was an indispensable source of research for me.

But I was part of a small technical minority, and over time Usenet declined in importance. Fewer and fewer people read and posted to it, and it lost its utility. Google bought DejaNews and even made it a tab on Google.com, but Usenet never went mainstream.

Now the blogosphere seems like the second coming of Usenet to me. Instead of 80-character courier posts and > quoting, we have rich-text HTML and images. There are subtle model differences, however, which should help protect the blogophere against the anarchy and spam that brought Usenet down. Instead of newsgroups with perpetual turf wars, posts are maintained per-author, and trackbacks allow threading, but with built-in spam resistance. But the question remains: is this going to be the domain a small, self-selected technical (or perhaps literary) elite, or will it have broader mainstream significance?

Perhaps its heresy to say this on a blog, but I run into a lot of blog-skeptics here in silicon valley (I even work with some :-). They have an instinctual resistance to the wild-eyed enthusiasm surrounding blog media, learned from years of watching supposed next-big-things skulk away into memory (sometimes after considerable investment), and dismiss blogs as a fad. Or at best, like Usenet, forever the domain of an insular group that won't ever crossover to a mainstream audience (evidence to the contrary notwithstanding.) Others may be taking a pragmatic view, seeing blogs simply as a cost-effective way to reach critical influencers, and thus make use of the blogosphere as a useful PR channel, regardless of whether blogs eventually "make it" as a mainstream media source or not.

Given Google's previous experience with Usenet, their powerful incremental search platform, and a healthy skepticism about embracing the Next Big Thing, they're likely pursuing a "smart follower" wait-and-see approach. If blogs and feed search do turn out to be critical, they'll be able to show up with a best-of-breed solution in short order.

But is this just another search application, or something bigger?

Unlike Usenet, hidden away on a network that only a few had access too, behind character-cell newsreading interfaces, blogs are indistinguishable based on appearance or use from other Internet media outlets. Users often can't tell the difference between a high-quality blog and a "mainstream media" news site. And there are lots and lots of blogs. This material is part of what a typical internet user will be exposed to and consume. Collectively, it will become the largest source of incrementally published content on the net.

Search Engines are Phone Books

Local media advertising is dominated by two heavyweights: the phone books and the newspapers. Search engines are phone books; they provide a table of contents to the web. The goal of a search engine is to be as objective as possible; if you enter "ibm" into Google and ibm.com isn't the first result, there's no subjective editorial judgment, it's just broken.

This is the most lucrative point to hit folks with advertising. Double-digit CTRs and high conversion rates are the norm. Searches for "saturn vue" or "sunnyvale dentist" are valuable; by typing those terms into a search engine, a user turns themself into a lead, often worth several dollars for a single click.

But phone books are boring. Nobody reads the phonebook for fun, and you can't advertise some things in the phone book. Nobody googles for lunch ("94303 cheeseburger"?) Sales of things people don't know they're looking for (tire sales, two-for-one pizzas), and marketing to demographics (e.g. advertising cosmetics to teenage girls) don't fit well with keyword advertising. For this, ad copy must be paired with dynamic content.

The Daily Internet

The proliferation of incremental content sources, all pumping out new material on a regular basis, is what the mainstream Internet user will consume. It's the difference between doing research or reading a magazine. At Topix.net we believe that editorial automation is necessary to manage this massive, growing content stream. Other startups like Feedster and Technorati are also focused on improving access to the incremental Internet. This is the future of audience on the net, as well as the next online advertising frontier. Rumors indicate Yahoo has something big in the works to embrace this shift, it will be interesting to see how MSN and Google respond.