Excellent article from Bob Garfield in AdAge on the subject of data mining + inspired analysis = serious monetization. (Hat tip to Scott Templar.)

Here’s a taste:

Volunteered data, priceless as it is, nonetheless takes a marketer only so far. To create a genuine bond, an intimate relationship, requires a thorough understanding of consumer behavior, consumer interests, consumer sentiments, consumer moods, consumer movements and so on — not the sort of information that you can put in a sign-up form, even if anybody were patient or generous or honest or self-aware enough to part with it. This requires what Sherlock Holmes called deduction. Also a bit of extrapolation, inference, intuition, divination, prediction and imputation. Or, put another way, guesswork.

[…]

“Let’s say you’d been on eBay three days ago and searched for a particular term,” [Matt] Ackley [ VP-net marketing at eBay] says. “We store that in the user’s cookie, so when we see that user out on the web again and we’re serving an ad — on Yahoo Mail, for instance — we’ll see that cookie. What we then do is pass that information to our banner ad. Now our banner ad is not a banner ad at all; it’s a miniature application. And what it does is then goes to eBay and finds items that are like that keyword and pools them into the banner ad.”

But beyond ad optimization, there is so much more going on. For instance, eBay algorithms intuit gender from the user’s first name and age from the shopping categories chosen.

“We know young people buy iPods and older people buy Longaberger baskets. This is the type of information you can tease out. Well, if you know somebody’s age and somebody’s gender and what kind of categories they’re active in, you can more or less predict what they might be looking for next.”

[…]

“We no longer have to rely on old cultural prophecies as to who is the right consumer for the right message. It no longer has to be microsample-based [à la Nielsen or Simmons]. We now have [total-population] data, and that changes everything. With [those] data, you can know essentially everything. You can find out all the things that are nonintuitive or counterintuitive that are excellent predictors. … There’s a lot of power in that.” [Dave Morgan, founder of Tacoda, the behavioral-marketing firm sold to AOL in 2007 for a reported $275 million.]

[…]

Reed Hastings, founder and CEO of Netflix, [describes] not only his company’s methods but also the essence of collaborative filtering, which is one of the “ABCs of predictive marketing.” B is behavioral: tracking your path online. C is contextual: paying attention to keywords, and A is associative: divining your tastes and interests based on patterns established by people like you.

If Netflix can figure out I admire “Manhattan,” “Strictly Ballroom,” “Happiness,” “The Girl in the Café” and “Downfall,” how badly can “Rabbit Proof Fence” and “Fitzcarraldo” disappoint me? The consequence is a great boon to me: easier selection process, fewer duds. It’s an obvious boon to Netflix, which had 239,000 subscribers when Cinematch was launched vs. 8.4 million today. And it is a veritable godsend to the movie industry — not to the Hollywood-studio part of the industry; “Spider-Man 17” or whatever will do just fine on its own. Netflix’s impact is on cinema’s everything else, the so-called Long Tail of moviemaking.

The Long Tail is the coinage of Wired Editor Chris Anderson, whose seminal 2004 magazine article on the subject yielded an ongoing blog about it, which in turn yielded a best-selling 2006 book about it — the “it” being how digital technology has ended the near-monopoly on distribution enjoyed since the Industrial Revolution by mainstream blockbusters at the expense of niche goods and services. The fat head of the Long Tail is “Spider-Man.” Way, way, wayyyy down in the skinny middle is “Fitzcarraldo.” But now I can rent them both in one click.

[…]

Compare prefiltering, then, to post-filtering — collaborative filtering — which, with the ultimate benefit of hindsight (it operates only in hindsight), knows everything. This is especially useful in a digital, Long Tail universe of seemingly infinite choices. Like my friends, former Soviet refugees, who walked into their first American supermarket and burst into tears, we are easily overwhelmed by the astonishing array of items on the internet’s virtual shelves. This phenomenon is often described as “information overload,” but Clay Shirky, author of “Here Comes Everybody” and professor of new media at New York University, says that’s not quite right. We suffer, he says, “not from information overload but filtering failure. The minute people are exposed to reality, they freak out. What collaborative filtering does is replace categorizations with preference.”

To truly appreciate the article in its full glory and start divining ways to apply some of the lessons learned here to your own business, go here.

Have a great Monday!