Online merchants such as Amazon, iTunes and Netflix may stock more items than your local book, CD, or video store, but they are no friend to “niche culture”. Internet sharing mechanisms such as YouTube and Google PageRank, which distil the clicks of millions of people into recommendations, may also be promoting an online monoculture. Even word of mouth recommendations such as blogging links may exert a homogenizing pressure and lead to an online culture that is less democratic and less equitable, than offline culture.
Whenever I make these claims someone says “Well I use Netflix and it’s shown me all kinds of films I didn’t know about before. It’s broadened my experience, so that’s an increase in diversity.” And someone else points to the latest viral home video on YouTube as evidence of niche success.
So this post explains why your gut feel is wrong.
The argument comes from a paper by Daniel M. Fleder and Kartik Hosanagar called Blockbuster Culture’s Next Rise or Fall: The Impact of Recommender Systems on Sales Diversity. They simulate a number of different kinds of recommender system and look at how these systems affect the diversity of a set of choices. Towards the end of the paper they observe that some of their recommender systems increase the experience of diversity for every individual in the sample and yet decrease the overall diversity of the culture. So I wrote a program that does basically what they do in their paper and tweaked it to highlight this result.
The result is what’s important here, rather than the particular algorithm used to generate this instance of it. But I know some people will want to know how the results are generated, so I’ll give a short sketch. If you want more than this, Fleder and Hosanagar provide details, my tweaks to their model are available as source code (python) if you want, and if you post in the comments we could get into a discussion. But it’s not important, trust me.
Each simulation starts with 48 customers and 48 products. Each product is described by two attributes, with values generated according to a normal distribution. So the products are distributed on a two-dimensional grid, with a value of about -3 to +3 along each axis. Each customer is assigned a taste for each attribute, so they also are scattered about in the same space. The idea is that a customer will prefer, other things being equal, a product that is close to it in these attributes. Here are two distributions of customers (blue) and products (red). You can see that most customers share a mainstream taste around the middle of the graph, but there are a few who have odd tastes off to the edges. Likewise, most products have attributes that are mainstream, but there are a few “niche” products closer to the edge.
In this particular simulation, a customer can choose the same item over and over again, so it simulates something like streaming radio more than a bookstore. Each simulation starts off with a priming phase, in which each customer makes 75 choices according to a function which favours nearby products, but with some randomness so that they may on occasion choose one further away. After 75 choices we turn on a recommender function. Whenever a customer goes to make a choice, the recommender system identifies a product and recommends it to the customer. The recommendation increases the chance that the customer will choose the recommended product. Fleder and Hosanagar look at a few recommender functions. The one I use works like this:
- The set of 48 customers is divided into equal-sized communities, with members chosen at random so they may not be close in taste.
- The recommender function chooses an item by looking at what customers in the same community have chosen. It recommends the one most popular among others in the community.
I’m just going to show you two simulations. Run 1 above – which I will call Internet World – treats the entire set of 48 customers as a single community. The other (run 28 above), which I will call Offline World, breaks it into 24 communities of two people each. In Offline World I will get recommendations from the people around me and you will get recommendations from the people around you, but these recommendations are separate and isolated. In Internet World we each get recommendations from all 48 customers.
Here are the results for the two simulation runs I’m going to focus on. The results of these simulations are far from the only possible outcome, but they show why the gut feeling may fail, and I’ve chosen them for that purpose.
In Internet World each customer experiences an average of 3.5 products over the course of 75 choices with an active recommender system, while in Offline World each customer experiences only 2.4 different products. So the wider set of people providing recommendations in Internet World has led to an increase in individual diversity. This is like saying that “Netflix shows me pictures I would never had heard about from my friends alone”, or “Amazon recommended a book I had never heard of, and I liked it”.
On the other hand, the overall diversity of the culture can be measured by the Gini coefficient of the products. A Gini coefficient of zero is complete equality (each product is chosen an equal number of times) and a Gini coefficient of 1 is complete inequality (only one product is ever chosen by anyone). And Internet World has a Gini of 0.79 while Offline World has a Gini of only 0.52. Internet World is less diverse than Offline World.
How can these seemingly contradictory results happen? Let’s take a look.
In the following graph, each dot is a customer, arranged in their two-attribute preference space (just like in the graphs above). But this time the area of each dot is proportional to the number of unique products they experience. So in Run 1 (Internet World) you can see that the dots are, on average, bigger than the dots in Run 28 (Offline World). This shows the greater individual experience of diversity in Internet World; for example, there is a customer with attributes of (1.1, -0.8) who samples no less than 38 different products, and only seven of the 48 customers stay with a single product throughout the whole simulation. Meanwhile in Offline World the most eclectic customer samples only nine and there are no fewer than 19 customers who sample just one product. The experience of individual customers in Internet World is of broader horizons and more selection, as recommendations pour in from far and wide, rather than from the limited experiences of their small community in Offline World. This picture has become the standard narrative of choice in the Internet World – our cultural experiences, liberated from the parochial tastes and limited awareness of those who happen to live close to us, are broadened by exposure to the wisdom of crowds, and the result is variety, diversity, and democratization. It is the age of the niche.
But wait!
Here is a graph of the products in each simulation. This time, the area of each dot shows its popularity: how often a customer chooses it.
You can see that on the left, in Internet World, a few products were chosen a lot, especially the one centred on about (-0.2, -0.2). In Offline World there are many more medium-sized dots, showing that the consumption of products is more equal. In Internet World one product has “gone viral” and gets chosen over 1500 times out of the total of 3600, while 26 products languish in the obscurity of being sampled fewer than ten times. In Offline World no single product is chosen more than 10% of the time, and only 14 products are sampled fewer than ten times. In short, niche products do better in Offline World than in Internet World.
While each customer on average experiences more unique products in Internet World, the recommender system generates a correlation among the customers. To use a geographical analogy, in Internet World the customers see further, but they are all looking out from the same tall hilltop. In Offline World individual customers are standing on different, lower, hilltops. They may not see as far individually, but more of the ground is visible to someone. In Internet World, a lot of the ground cannot be seen by anyone because they are all standing on the same big hilltop.
The end result is the Gini values mentioned before. Here are Lorentz curves for Internet World (blue) and Offline World (green), in which the products are lined up in order of increasing popularity along the x axis, and the cumulative choices for those products is plotted up the Y axis.
So there it is. Individual diversity and cultural homogeneity coexisting in what we might call monopoly populism.
But don’t think this is just about automated recommender systems, like the ones that Amazon and Netflix use. The recommender “system” could be anything that tends to build on its own popularity, including word of mouth. A couple of weeks ago someone pointed me to this video of Madin, a six-year-old soccer prodigy from Algeria, and the next day my son, who moves in very different online circles to me, was watching the same one. I know who Jim Cramer is even though we don’t get CNBC in Canada because everyone is talking about him and helping his disembodied head to shoot down Jon Stewart. More people watched Tina Fey being Sarah Palin online than on Saturday Night Live, and Fey is now famous in countries where no one watches the TV show. Clay Shirky writes an essay and I get five different links to it in my Google Reader feed in one morning. Our online experiences are heavily correlated, and we end up with monopoly populism.
A “niche”, remember, is a protected and hidden recess or cranny, not just another row in a big database. Ecological niches need protection from the surrounding harsh environment if they are to thrive. Simply putting lots of music into a single online iTunes store is no recipe for a broad, niche-friendly culture.
Are you familiar with eurytopy and stenotopy in ecology? In the Amazon forest, the climate is steady enough that a species can specialise in a small microclimate that does not move or change very much, and individuals in that species can be *stenotopic*, and survive without being able to move very far or adapt to a different location. They become the world’s best organisms for existing in that exact spot, and another species a few miles away is the expert in another spot.
The polar tundra is different. You’d think the lower energy levels would lead to less biomass, but that doesn’t seem to be much of an issue (the nearby polar oceans are famous for their fecundity). What is a bigger issue is the variation in conditions through the year. An organism that wants to survive must be *eurytopic*, either good at living in all the conditions that apply in that spot, or able to move to follow the seasons. Either option makes it sort of an expert at living in almost any place, so it isn’t excluded by a neighbour who’s a better local expert.
The result is that, although any given square mile in the tundra probably has as much variety as a square mile in the tropics (if not more), it’s the same variety in every square mile in the tundra, and a different variety in every square mile of the forest.
This sounds like the situation you’re describing in the mobile world wide web (Thomas Friedman’s ‘Flat World’) compared with the old lumpy world of slowly-travelling information: greater diversity of goods for the mean customer, but less diversity of customers for the mean good.
PS I thought you were using “Clay Shirky” as a for-instance, but I got the link to, and read, his amazing take-down a few minutes after writing the comment above 🙂
I’ve never heard either word. Fascinating comment, and the parallel does seem very close.
Cheers,
Tom
It was a little random. I have mixed feelings about Shirky – a good writer and very provocative but wanders into techno-utopianism from time to time. Not in that essay though.
Interesting article. I’m a bit puzzled about the choice of recommendation algorithm used for your argument though:
* The set of 48 customers is divided into equal-sized communities, with members chosen at random so they may not be close in taste.
* The recommender function chooses an item by looking at what customers in the same community have chosen. It recommends the one most popular among others in the community.
Surely with the assumptions made this algorithm would “obviously” (in its loosest sense) lead to this result. Isn’t this effectively the same algorithm used by readers of the “Top 10 Downloads” and such lists? Or have I missed something in the algorithm description?
A more likely algorithm would be based on communities of people clustered based on similarity of taste. (e.g. split taste dimensions into a grid, communities are people living in the same grid rectangle). People would then only get recommendations based on people with similar tastes.
The “downside” of such an algorithm is that existing tastes/prejudices are simply reinforced.
The simple 2-D point preference model appears to be a real limitation here. A better assumption might be that people have a number of 2-D preferences, but initially only know about one of them. (“I like 1970’s rock music, but don’t know (yet) that I would like Gregorian Chant”).
Will a recommendation system enable me to discover my unknown preferences as well as identifying new material that conforms to my known preferences? And will such a system have any undesirable side-effects such that material which I might like is not produced?
There is something obvious to the fact that basing a recommendation on popularity and pooling from a big crowd leads to lack of diversity. What is not obvious – to me anyway – is that this decrease in overall diversity can coexist with an increase in individual experience of diversity.
There are all kinds of recommendation systems and it is easy to design one that promotes diversity (recommend an item that few people have viewed and which you have not recommended before). But while the original paper is about recommender systems and their design, that’s not really the point here. So while your questions are good ones about whether good recommender systems could help you discover new tastes, I think you are exploring topics beyond what I’m doing here.
Yes I agree that the decrease in overall diversity coexisting with an individual experience of diversity is a surprising result.
What concerns me is that if the model of personal preferences (an unchanging 2-D point) is too simplified, or the choice of recommendation algorithm too careless, then the results might not be valid outside of the model.
But as you correctly point out, what makes a good recommender system is a different issue.
I think we need to look at a weighted value of the recommendation. Customers who are closer in preference to me have a higher weight than those who are farther away in preference. Think aboutwhen trying a new restaurant. You might consider trying a place a trusted friend (or a foodie) suggested you try, but you might actually be turned off by the recommendation of some crazy idiot you have nothing in common with. I’m NOT going to select this product b/c the last guy to buy it was (-3,-3) and that’s just crazy land.
I get the feeling that you’re on to something here, that the internet as a whole is eliminating smaller niche products and helping to really grow special niche products with a sort of populism. However, I think that given the niche, there are still strong relationships to be made based on the quality of recommendation and the value it holds to the customer.
I also question what is happening as the internet becomes more individualized. As I’m able to start receiving recommendations from just my facebook friends, or twitter followers, or whatnot, it seems to reason that overall diversity may reduce but individual niche areas may grow larger. Not everything comes into mainstream, but the mainstream becomes a lot broader and more diverse. Thanks for the interesting models.
The basic result is similar to the work Paul Krugman got his Nobel prize for. His work on comparative advantage showed that under some simplifying assumptions, global trade can produce a situation where every individual sees an increase in product diversity while globally the number of products available shrinks.
And basically, that’s what you’re showing here. By providing consumers with a larger marketplace, each individual sees more diversity, but the underperformers die off.
each individual sees more diversity, but the underperformers die off
a) “Under”? “Perform”? That’s a bit like saying libraries should throw out collections they’ve accumulated over decades because those books aren’t being borrowed as often as more recent acquisitions. (Oh, wait, that’s what they are doing.)
b) Someone lend me a time machine, I want to stop Huxley being conceived.
So if I’m someone with a niche product, how do I stop myself from dying out?
The objective it seems regards Amazon and Netflix is to get into those top 10, top 50, top 100 listings otherwise I’m going to lanquish in obscurity.
I’m guessing that I can
(a) ask my network to recommend my product – to bump me up the list and hope that new consumers recommend me – and/or
(b) I can buy my way to the top by buying “popular” products and then buying my own so that I get a “people who bought this also bought this”.
I know that people who first subscribe to Netflix will first watch what’s popular now, then they’ll watch all the “classics” and then they’ll start to look for undiscovered. I’d guess it would be the same with iTunes but as it’s not an all-you-can-eat model consumers are likely to be even more selective and less willing to risk something. My point in this paragraph is how important it is to get listed as a “new arrival” because this could be the only chance a niche product gets for exposure. Unfortunately it looks like iTunes is far too selective about what it lists as a new arrival and tends towards all big titles.
The whole construct rests on implicit assumption that moving from 48 customers and 48 products to millions of customers/products spread over multitude of social strata will not introduce factors rendering the entire thesis incongruous.
That’s how all these macro-economists keep getting Nobels while the real economies keep veering and swerving into directions none of them can predict.
Isn’t it fascinating, for example, that above mentioned Krugman’s work is based on “some simplifying assumptions”? As we perfectly know from our daily experience, global trade indeed already produced abundance of situations where the number of products available shrank.
Nifty thoughts. What is the value? Are we trying to figure out how to exploit market niches, how to be more diversified or how to be more unified? As a pure search for truth, this is very provocative. As a method to figure out how to turn lead into gold or make it rain, it is probably dangerous.
A few questions:
1) Why do these two scenarios have different starting points?
2) Is the recommender system’s recommendation stable throughout the 75 iterations? How many iterations until it becomes stable?
3) How sensitive are your results to the particular parameters you chose for:
a) the variance of the customer’s individual choice [you say they are “more likely” to choose the closest product but how much more likely?];
b) recommendations based on the single-most-popular product in the community instead of based on a sample drawn from the 10-most-popular products in the community;
c) the weight given by each individual to the recommendation of the community;
d) the use of a “global” community rather than “targeted” communities of individuals chosen based on similar tastes;
e) the number of iterations run before turning on the recommendation system?
In my opinion many of these could have a significant effect. It is unclear how meaningful an isolated result is without some parameter sensitivity analysis.
mk – I’m not trying to produce a prediction that a particular model will always generate the kind of outcome I talk about. If you want to see it done right, see the Fleder and Hosanagar paper I link to above. But even with a fully-calibrated model, there would be the question of what real recommender system (if any) it simulated, and of course there would be lots of room for vagueness there. My point was simpler, that at the crude end of the discussion personal experience is not necessarily a guide to what’s happening to the overall diversity of culture. Also, I do think that there are places on the Internet where the kind of coordinated recommendations happen – like iTunes for example, where we all see a very similar front page. But I can’t get at the proprietary iTunes numbers to verify, of course.
Zenberg – I’m more interested in the shape of the culture that the Internet generates rather than business models.
Ozornik – I believe we’ve had the discussion at marginalrevolution, so I’ll leave it there.
Thanks for the comments, everyone.
It seems like there are three big gaps in your experiment. The first is what was suggested by mk–if it’s just the recommender that causes the different results, use the same starting positions for all products and customers, or run lots of random runs for each. Just picking two examples is not meaningful in the slightest.
Second, as several people mentioned, how can you be sure that there isn’t a recommender which improves both individual and global diversity. If there were such a recommender, that would really be something, and non-random clustering seems likely to lead to one.
And finally, why do you hold constant the number of products that a customer tries over the the duration of the simulation? It’s not that I watch the movie Netflix recommends instead of watching my favorite movie every weekend. If Netflix doesn’t make an enticing recommendation, I don’t watch any movie at all. If recommenders cause more total products to be consumed, it is very possible for both global and individual diversity to improve.
I’m not trying to reduce the whole internet to a single 48-customer simulation. I’m trying to highlight a mechanism that can be at work when a system aggregates many different opinions into a single recommendation. And to identify mechanisms you have to use simplified models. So I can’t “be sure there isn’t a recommender which improves both individual and global diversity”. In fact I’m sure there is. It would be easy to construct one (recommend a product that (a) has been chosen rarely, and (b) that been recommended rarely).
It is true that “If recommenders cause more total products to be consumed, it is very possible for both global and individual diversity to improve.” Right now I don’t see that happening – but that’s an overall impression, not a result of this particular little simulation.
“Each product is described by two attributes, with values generated according to a normal distribution. So the products are distributed on a two-dimensional grid, with a value of about -3 to +3 along each axis. Each customer is assigned a taste for each attribute, so they also are scattered about in the same space.”
How do you determine the distributions here? It looks to be a normal distribution. What does it look like when you do something like a log normal distribution?
Your work here is very interesting, and I agree with the premise and results. My gut is that the application of normal distributions will continue to fade.
I don’t have the education to have a discussion with any of you on the algorithms or statistical distribution; but I would simply like to relay to you some personal observations.
Firstly, thank you for the article – it is a subject I have been thinking and talking about (without the science you include) for some time to all that would listen. Years ago, when I first saw recommendation systems on the net or aggregation websites (/. or digg or reddit) I innately felt that these systems would provide “gravitational” pull to certain subjects or objects and lead towards a monoculture. I still feel this way today. Sure there are a number of discovery systems being introduced but even these ones allow for weighting and as such are likely destined to become echo chambers. I suspect that these systems are creating islands of interest and at the same time decreasing mobility (from one island of interest to another).
To make a long story short: it isn’t the end of niche – but the dying days of serendipity. I personally fear this. For me “serendipity” is the mechanism by which “I know how much I don’t know”. It helps me to be humble about my understanding of the world, which in turn allows me greater freedom to explore solutions to problems (even in deciding that a problem may not be a problem at all). Does this mean that I am less competitive than those that quickly reach for the recommended tool and execute the recommended procedure? I suppose – yes. Thus, the individual needs to make a proper decision: when to ascribe to “common knowledge” and when to allow serendipity to place you in a some random location. Has this made my life better – I don’t really know… but my gut instinct is that randomness is good, and needs to be actively preserved. Thanks again for the article.
Just this morning I was thinking that the sites/blogs that I read regularly were, more and more, exchanging hat tips for finding interesting things.
then I come across the link to this article on an aggregator that I visit somewhat regularly. Of course they gave a hat tip to a blogger that I read somewhat regularly.
Your result is very interesting (I already teach that paper in class) but avoids two essentials elements :
* What reduces global diversity is the existence of a common institution, not “the Internet”. In many coutries (USA excluded) National News has been so important that any diversity offered by Web sites appeared as a new and welcome departure from monoculture; more generally, it’s the distribution of news sources off- and on-line that you need to compare (and how connected they are).
* Algorithms offer the possibility to decide whether to encourage niche or global success: Amazon makes more money, because more click-through, by suggesting Harry Potter every time, but they don’t; it leverages their uniqueness and encourages reader’s curiosity. Physical insitutions have the same recommendation for everyone, and cannot offer diversity in any other way then by being many.
I’ll put more details on my blog soon: twocroissants.wordpress.com
It seems like this is actually an answer to those who claim that the Internet will lead to an atomization of culture, with everyone following their own narrow and insular areas of interest. This suggests that, given a choice and the information to make that choice, people will tend on their own to gather around a common cultural experience.
What a very nice little dish pan model <!”>http://web.mit.edu/krugman/www/dishpan.html>! I suspect that you can show that buyer satisfaction is reduced by the arrival of this recommending intermediary as well.
Of course buyer preferences product features are very messy. Buyers are usually very poor at knowing their own preferences. Their preferences are extremely pliable. If they do they can’t map them into the nomenclature of the seller. It’s part of the sales/shopping problem to shape the consensus that this exactly what your really wanted.
I love how it helps me think about the buyer-product matching problem. It is no surprise that any intermediary would effect the outcomes – for better and worse. The effect here is that the recommendation system get’s jammed by the products near the avg. of the distribution; and then get’s swept up in a network effect.
A recommendation system that was actually modeling the buyer and product place in the feature space might temper that somewhat. But that’s obviously extremely hard.
Really interesting post and really interesting comment by Derek.
So for a given event, all the newspapers take the exactly same set of “stenotopic” pictures to illustrate their article even if 100s are available (Except “The big picture” http://www.boston.com/bigpicture/ ).
I have heard that the Apple AppStore suffers hugely from this problem of getting on the front page. And I agree with your point about subscription models versus pay-by-the-piece – I suspect it makes a big difference.
Films and books and tunes are all very different though, so I don’t know how you make progress with a particular case – it’s the same struggle it’s always been – but there’s still always a chance.
I’ve definitely discovered some great books because they were alphabetically next to others in the library. I’m all in favour of randomness too.
I’ll watch out for that. Cheers.
One thing I’ve not sorted out is whether to trust the assumption of it being a matching problem at all. See http://www.nytimes.com/2007/04/15/magazine/15wwlnidealab.t.html for why it might not be.
I’m not the only one who had never heard these worlds. Neither had Paul Kedrosky.
Here’s the thing, most recommendation systems rely heavily on product attribute similarity – not just popularity or community preference. It seems to me that both this model and the Fleder Hosanger models discount this fact – and it is a huge one at play in how recommendation systems work all over the web.
Pandora for example, makes recommendations based upon how well the “musical dna” (product attributes) of a song match the “dna” of any other. A song (product) recommendation, then, is *not* made based on the user’s preference similarities to others on the web (the community), but on the similarity of the product itself to the amazingly diverse long tail of other products. So in this case, we’re seeing pure diversity discovery unbiased by community viral popularity.
Is it an oversimplification to say, sure, the frictionless nature of information discovery on the web makes it possible that we all are “aware of the popular stuff” – but that certainly hasn’t reduced our ability to simultaneously discover (and ultimately consume) more from the long tail, right?
Alot has been said in the past about ‘cumulative disadvantage’ in the context of web 2.0 and a more socially focused web. Here’s some of my thoughts from a few years ago:
http://www.kurtvoelker.com/items/view/325/cumulative-dis-advantage
The best thing I can imagine for increasing diversity is:
A) a “Show me a random thingy” button
along with
B) some way of rewarding the people who viewed and recommended an item before it became popular.
Kurt – Interesting thoughts, but I disagree.
I think Pandora is unusual in its musical dna approach, and the idea of building attributes into a system is perhaps something limited to music. For example, the leaders in the Netflix Prize competition are using nothing about a movie/DVD except its title and release date – everything else comes from viewer assessments of movies. And Amazon doesn’t build any attributes into its system either so far as I know.
As a result, in the movie and book space at lease, products don’t have attributes until people rate them. This is the “cold start” problem that some recommender system people are looking at.
The Watts study that you don’t like highlights the uncertain nature of products having well-defined attributes as well. It shows that people’s perception of one song or another are shaped by others recommendation. In the last twelve months of the Netflix Prize one of the new factors the leading teams are building into their approaches is to take the date of ratings into account – the attributes of some movies apparently change over time.
Well I haven’t done much thinking about actual constructive ideas. I’m more interested in pouring cold water on others :). But I like (B) a lot.
Interesting post.
Your idea about the internet also relates to island biogeography theory in ecology. The same amount of land in a bunch of small islands will have more species than one big island. This theoretical arguement is one reason why ecologists are worried about the increased transport of organisms around the world coupling the world together (i.e. biotic homogenization/invasive spp) making all the islands into one big island which is able to support much less diversity.