The Long Tail 10 – The Paradise of Choice

This is another part of my critical reader’s companion to The Long Tail, and it discusses Chapter 10 – The Paradise of Choice. Part 0 is here. You can find a complete list of the Long Tail pieces here.

The purpose of this chapter is to counter "the notion that ‘too much choice’ is overwhelming, a belief so common and ill-founded that it deserves its own chapter" [167].

The last 50 years have seen "an explosion of variety" [169], which Anderson ascribes to three factors. The first is "globalization and the hyperefficient supply chains it brings"; the second is the change in the population – in the US, he mentions ethnic diversity and increasing affluence, leading to a cultural shift from "I want to be normal" to "I want to be special"; the third is The Long Tail – or the proliferation of variety brought about by iTunes, Netflix, Amazon, and eBay.

But there’s variety and then there’s variety. Grocery shopping is often a focus for this kind of discussion (and the book uses it in the next section) so let’s use supermarkets as a way into the topic. For me, the proliferation of options at the cheese counter is a great thing: the range of options there seems to me a real variety. On the other hand, the toothpaste section (over forty different types of toothpaste) does not seem to be real variety. Now this could just be me, who likes cheese and can’t really be bothered about toothpaste as long as it does the job, but I think it’s more than that. There is another difference between the two, and it’s the issue of manufacturers or suppliers. The cheese counter contains products from a variety of producers, while the toothpaste shelves contain products mainly from two producers – Colgate-Palmolive has 37% of the American market; Procter & Gamble (makers of Crest and now merged with Gillette) follow with 34% – with a few "smaller" players like GlaxoSmithKline (Aquafresh) and Unilever filling out the remainder. If anything, the toothpaste market has become more dominated over the years by this small number of companies; meanwhile, the profusion of items on the shelf is growing. Oligopoly Watch (from a few years ago)  describes these twin trends of corporate consolidation and product profusion in the world of toothpaste. Is the combination an increase or a decrease in real "choice" and "variety"?

Anderson is as ebulliently optimistic as ever that the increase in number of products is simply a reflection of what we consumers want – "More choice really is better" [174], but very well-known research by Richard Schmalensee showed that the proliferation of toothpaste varieties may have little to do with responding to consumer demand, and be more about preventing other companies from getting a foothold in the market. Back in 1978 he looked at the ready-to-eat breakfast cereal market, where six companies dominated the market and introduced eighty different brands between 1950 and 1972 – more choice, for sure. Now Schmalensee is no radical – he was on Microsoft’s team in their anti-trust suit, arguing that Microsoft is not a monopoly – but he concluded that this particular case of product proliferation was all about trying to deter entry by other companies. The strategy is to "pack the product space and leave no profitable niche unfilled" (this from the standard graduate text by Tirole, p. 346). Schmalensee showed "how a cartel (a group of firms that act as a single monopolist) crowds a product space". The cartel in this case needs no explicit collusion, but the strategies of the incumbent firms complement each other to produce the same effect as a monopoly. The result – higher prices for consumers, as the incumbent firms can raise prices without new competitors coming to undercut them.

How much of the "explosion of variety" that we have experienced is real variety, and how much is entry deterrence and the ability to raise prices? It requires a lot more space and expertise than I have to pick apart what’s what in the myriad of different retail spaces, but it is something to think about when standing in front of a wall of shampoo, deodorant, mustard, breakfast cereal, or whatever. One would think that if The Long Tail is, among other things, an "economics research project" [11] and "a preview of 21st century economics" [11] then its author might have looked a little harder at the work of actual economists, however 20th century they might be.

Too Much Choice? [170-172] is a quick summary of one of the points made in Barry Schwartz’s excellent book The Paradox of Choice. It describes one well-known experiment that Schwartz talks about on pages 19-20, carried out originally by Sheena Iyengar of Columbia University. The experimenters set up a table at a specialty food store and offered customers a taste of a range of jams, and a $1 coupon to use against the purchase of a jam. When only 6 jams were put on the table, 30 percent of customers went on to buy a jar of jam; when 24 jams were put on the table, only 3 percent did. Faced with extra options, people shied away from choosing any of them. After presenting the experiment, Anderson goes on to say:

Schwartz describes the conclusion this way:

As the number of choices keeps growing, negative aspects of having a multitude of options begin to appear. As the number of choices grows further, the negatives escalate until we become overloaded. At this point, choice no longer liberates, but debilitates. It might even be said to tyrannize.

I found those sentences on page 2 of the book (The Long Tail does not indicate where they came from – it’s endnotes are lousy). They are separate from the jam experiment (pp 19-20). Schwartz’s book addresses a broad range of questions about the proliferation of choices that we are faced with, and while the sentences do reflect an argument that Schwartz makes, it oversimplifies his reaction to this particular experiment.

Anderson’s response to Schwarts is to say:

I’m skeptical. The alternative to letting people choose is choosing for them. The lessons of a century of retail science (along with the history of Soviet department stores) are that this is not what most consumers want [171].

His rebuttal does not do justice to The Paradox of Choice. Two sentences before the convenient quotation above, Schwartz says "When people have no choice, life is almost unbearable", and a few sentences after them he says "Choice is essential to autonomy, which is absolutely fundamental to well-being. Healthy people want and need to direct their own lives". So Schwartz is clearly not advocating Soviet department stores here, or the elimination of personal choice. The suggestion that the only options are "letting people choose" or "choosing for them" is a silly false dichotomy that I got accused of with my book too. There are many forms of choice, from individual consumer choice to our choices as citizens to collective choices – not just "choosing for them". Anderson is substituting a quick dig for real argument. It’s weak. But then, Schwartz’s book is detailed, subtle, and packed full of references to work by other people…

In fact, Schwartz’s book is more about how to cope with the profusion of choice than advocating a removal of it. One of eleven recommendations he makes is to "satisfice", but Anderson ignores the other ten and says: "Schwartz recommends that consumers ‘satisfice,’ in the jargon of social science, not ‘maximize’. In other words, they’d be happier if they just settled for what was in front of them rather than obsessing over whether something else might be even better" [171]. Contrast this with Schwartz’s actual definition of "satisfice" (p78):

To satisfice is to settle for something that is good enough and not worry about the possibility that there might be something better. A satisficer has criteria and standards. She searches until she finds an item that meets those standards, and at that point, she stops.

I don’t think Anderson’s paraphrase is a fair one, and I don’t think his arguments refute much of The Paradox of Choice. (They also don’t even start addressing most of the problems that are the subject of No One Makes You Shop at Wal-Mart, but that’s another story).

Variety is Not Enough [172-175] starts out by refuting Iyengar’s jam experiment. Anderson wrote to Iyengar and asked "why the people who should actually know the most about consumer choice in a supermarket were ignoring their [Iyengar and coworkers’] conclusions" [172]. He summarizes her reply (and a work in progress [PDF download]) as "The solution, they found, is to order the choice in ways that actually help the consumers" [173]. And with this, Anderson’s faith in limitless choice restored and he is off to find out how online resellers manage to guide people through their choices.

But just as Schwartz is more nuanced than Anderson gives him credit for, so Iyengar and coworkers are more nuanced in their conclusions. They distinguish "Preference Constructors" from "Preference Matchers". Preference Matchers are often experienced with the particular choice they are faced with (choosing a cup of Starbucks coffee is one they discuss); they are "someone who enters the decision making task looking for either a specific option or an option that possesses a number of pre-defined attributes. This type of chooser restricts the multitude of available options in a purchasing situation by creating “consideration sets,” or sets of options with a high probability of containing his or her optimal choice". Meanwhile "Preference Constructors" are those approaching the choice for the first time. Faced with a dizzying array (24 jams) such a chooser may shy away. They can be helped by providing some guidance: "To enable Preference Constructors to distinguish a hazelnut steamer from a vanilla chai and thus successfully form their preference criteria before identifying their preferred option, these decision makers may require more than a limitless, unstructured choice set." Here is the basis for Anderson’s optimism.

His contention is that online retailers are ideally placed to guide you through the choice in a comfortable way, so that us consumers are not faced with the 24-jam problem. But I’m just an old cynic. Whereas Chris Anderson is thankful for all this guidance he’s getting, I’m just suspicious. As George Akerlof wrote (and he knows about people guiding you through choices), the problem goes back as far as horse trading: "“if he wants   to sell that horse, do I really want to buy   it?” Such questioning is fundamental to the market for   horses and used cars, but it is also at least minimally present   in every market transaction."

Here’s Anderson’s take on how online retailers can guide people through a proliferation of choices.

There are a nearly infinite number [nitpick alert: "nearly infinite"??? – TS] of techniques to tap the latent information in a marketplace and make that selection process easier. You can sort by price, by ratings, by date, and by genre. You can read customer reviews. You can compare prices across products and, if you want, head off to Google to find out as much about the product as you can imagine. Recommendations suggest products that ‘people like you’ have been buying, and surprisingly enough, they’re often on-target. Even if you know nothing about the category, ranking best-sellers will reveal the most popular choice, which both makes selection easier and also tends to minimize post-sale regret. After all, if everyone else picked a given product, it can’t be that bad. [173]

I confess I did a double-take when I read that paragraph. The first half is confusing: is the goal of "supplementing an unstructured choice set" really accomplished by adding to that bewildering array a second bewildering array of different techniques to find out bits and pieces about the merits of different choices? I don’t think so. And the second half is a concise argument as to why online retailers so often fail to drive demand down to The Long Tail – in direct contrast to what he claims elsewhere. Anything that gets people to choose an item just because other people have chosen it leads to increasing returns. Once you’ve chosen it, it will be recommended to other people because you bought it, and so on. Note that without recommendations, you can’t even tell if they liked it or not once they bought it. And pointing people to best sellers because "it can’t be all that bad"? This is the logic of the blockbuster. Nevertheless, Anderson is so pleased with his conclusion that he reiterates it three times: "More choice really is better… The paradox of choice turned out to be more about the poverty of help in making that choice than a rejection of plenty. Order it wrong and choice is oppressive; order it right and it’s liberating" [174].

The other argument that Anderson has in favour of unrestricted choice is taken from libertarian journalist Virginia Postrel, and is again about help. It is true that, around most choices that are both expensive and complex, an industry of brokers rises up. Real estate agents, wedding planners, financial planners and so on, all "help us be ourselves" [174]. Another way of looking at such an industry is that it represents the cost of choosing (wedding planners are not free, after all): the rise of such an industry indicates that there is a cost to choice that we are prepared to pay substantially for, but this is in itself neither an argument for or against the proliferation of variety and pseudo-variety that we are immersed in.

The Economics of Variety [175-176] finishes the chapter by asking "Does more choice encourage consumers to buy more?" [175], and concluding that, by making choices easier, digital distribution "widens the field of possible customers and shortens the search time. Over time, this should increase sales and grow the overall market. As we saw in Chapter 8, longer tails can be thicker too" [176].

I’m not entirely sure what he means here by "the overall market", but if he is claiming – as I think he is – that the overall economy will grow because people will buy more stuff, then this is a claim that needs a lot of substantiation. The money, after all, has to come from somewhere. Unless he is advocating a further increase in consumer debt as a way of boosting the overall economy, then I’m not sure what his argument is.

But then that’s how I feel about the whole book, of course. The Long Tail sounds nice, and it would be great if online retail could not only break the tyranny of the blockbuster and promote a diverse niche culture, but also boost the economy at the same time – but just because it would be nice doesn’t make it so, and the book continues to fail to convince.