Predicting hits may be like predicting the weather

In the New York Times sociologist David Card physicist turned sociologist Duncan Watts writes about how cultural hits may be, like the weather, impossible to predict. It comes down to how much we like stuff because it’s good and how much we like it because other people like it. If it’s the latter, then it becomes impossible to predict at some point. What’s nice is that they did some experiments to demonstrate some of this in the lab too – you’ll have to click through to see it.

A side effect of this would be that there is a strong limit to the effectiveness of recommendation schemes from Amazon and Netflix and so on. Maybe our preferences are just too damn quirky to be captured, no matter how fancy the algorithm.

Here’s a few paragraphs from the article.

Conventional marketing wisdom holds that predicting success in
cultural markets is mostly a matter of anticipating the preferences of
the millions of individual people who participate in them. From this
common-sense observation, it follows that if the experts could only
figure out what it was about, say, the music, songwriting and packaging
of Norah Jones that appealed to so many fans, they ought to be able to
replicate it at will. And indeed that’s pretty much what they try to
do. That they fail so frequently implies either that they aren’t
studying their own successes carefully enough or that they are not
paying sufficiently close attention to the changing preferences of
their audience.

The common-sense view, however, makes a big
assumption: that when people make decisions about what they like, they
do so independently of one another. But people almost never make
decisions independently — in part because the world abounds with so
many choices that we have little hope of ever finding what we want on
our own; in part because we are never really sure what we want anyway;
and in part because what we often want is not so much to experience the
“best” of everything as it is to experience the same things as other
people and thereby also experience the benefits of sharing.

There’s
nothing wrong with these tendencies. Ultimately, we’re all social
beings, and without one another to rely on, life would be not only
intolerable but meaningless. Yet our mutual dependence has unexpected
consequences, one of which is that if people do not make decisions
independently — if even in part they like things because other people
like them — then predicting hits is not only difficult but actually
impossible, no matter how much you know about individual tastes.

The
reason is that when people tend to like what other people like,
differences in popularity are subject to what is called “cumulative
advantage,” or the “rich get richer” effect. This means that if one
object happens to be slightly more popular than another at just the
right point, it will tend to become more popular still. As a result,
even tiny, random fluctuations can blow up, generating potentially
enormous long-run differences among even indistinguishable competitors
— a phenomenon that is similar in some ways to the famous “butterfly
effect” from chaos theory. Thus, if history were to be somehow rerun
many times, seemingly identical universes with the same set of
competitors and the same overall market tastes would quickly generate
different winners: Madonna
would have been popular in this world, but in some other version of
history, she would be a nobody, and someone we have never heard of
would be in her place.