Netflix Prize: Was The Napoleon Dynamite Problem Solved?

  By

I just gave a talk at work on “Recommender Systems and the Netflix Prize”, and included the two major popular articles about the prize in its final year or so. One was in Wired Magazine and one was in the New York Times., and each focused on one outstanding problem that the competitors faced. Wired looked at the quirkiness of users as they rate movies, and the NYT focused on the difficulty of predicting ratings for a handful of divisive movies.

Now that the contest is over we can answer the question, “were those problems solved?”

Let’s start with the Wired article. Entitled “This Psychologist Might Outsmart the Math Brains Competing for the Netflix Prize” [link] it interviewed Gavin Potter, aka “Just a guy in a garage”. Here’s the hook:

The computer scientists and statisticians at the top of the leaderboard have developed elaborate and carefully tuned algorithms for representing movie watchers by lists of numbers, from which their tastes in movies can be estimated by a formula. Which is fine, in Gavin Potter’s view — except people aren’t lists of numbers and don’t watch movies as if they were.

Potter is focusing on effects like the … Continue reading

Why I Am Such an Infrequent Blogger

  By

“My assumption, always, is that everyone knows everything I know AND MORE. Rephrase. Everyone who is interested in the kinds of thing that interest me knows everything I know AND MORE. If they're not interested they don't know but don't want to. So there's no point in mentioning things that strike me as interesting, unless a) these are events in the last, say, 5 minutes (so those disposed to be interested might not be au fait or b) I'm up for proselytizing (those not disposed to be interested might be with enough encouragement).”

See? Helen DeWitt even knows more than I do about why I am such an infrequent blogger.

Continue reading

More Long Tail: Everyone Is Still Wrong

  By

Every now and then another study comes out about the long tail and gets discussed in the usual places. More often than not, the end result is additional confusion, because the thing they are talking about (Chris Anderson’s book) defines the concept in many different ways, depending on what the author feels like talking about at the moment.

The latest is a working paper called Is Tom Cruise Threatened: Using Netflix Prize Data to Examine the Long Tail of Electronic Commerce, by Tom Tan and Serguei Netessine of the Wharton Business School at the University of Pennsylvania [PDF, summarized here]. It got slashdotted here, written up in The Register here, and Chris Anderson responds to it here.

Here is what the paper does. It  takes the sample of 100 million DVD ratings provided by Netflix for the recently-completed Netflix Prize, and breaks down the trends in ratings from 2000, when Netflix stocked relatively small numbers of titles and had relatively few users, to 2005 when Netflix had many more DVD titles and many more users. Then they ask whether demand for “hit movies” and “niche movies” increased or decreased over that time, as reflected … Continue reading

Google Book Settlement: I think I opted out

  By

This morning I opted out of the Google Book Settlement. At least, I think I did. After going through the forms and clicking the buttons there is a generic page saying something like "your opt-out has been received".  But I have no receipt, no acknowledgement – in short, no proof that I have opted out. Very odd. I emailed Rust Consulting (the settlement administrator) and assume I will get something sometime, but it seems very amateurish for what they keep telling us is such an important change.

As for why I opted out, well I hope to write about that. But I have a deadline for next weekend and a busy time at home (never mind the day job), so I can't do that right now, and this blog will continue to be quiet for a while.

Update: After emailing the settlement administrator, I did get an email confirming my opt-out.

Continue reading

No Free Stuff Here

  By

A few people have asked me whether I’m going to repeat my Critical Reader’s Guide to The Long Tail with Chris Anderson’s new book Free. It’s nice to be asked, so thank you. And I am, of course, susceptible to bribes and flattery. But the answer right now is No. Three reasons:

  • I don’t have a vendetta against the guy, so there’s no particular reason to go after him again.
  • To be honest, I doubt that it will be interesting enough to spend that amount of time on.
  • I don’t think the ideas will be as destructive as I think the Long Tail has been.

Continue reading