Author: Tom Slee

Online Monoculture and the End of the Niche

  By

Online merchants such as Amazon, iTunes and Netflix may stock more items than your local book, CD, or video store, but they are no friend to “niche culture”. Internet sharing mechanisms such as YouTube and Google PageRank, which distil the clicks of millions of people into recommendations, may also be promoting an online monoculture. Even word of mouth recommendations such as blogging links may exert a homogenizing pressure and lead to an online culture that is less democratic and less equitable, than offline culture.

Whenever I make these claims someone says “Well I use Netflix and it’s shown me all kinds of films I didn’t know about before. It’s broadened my experience, so that’s an increase in diversity.” And someone else points to the latest viral home video on YouTube as evidence of niche success.

So this post explains why your gut feel is wrong.

The argument comes from a paper by Daniel M. Fleder and Kartik Hosanagar called Blockbuster Culture’s Next Rise or Fall: The Impact of Recommender Systems on Sales Diversity. They simulate a number of different kinds of recommender system and look at how these systems affect the diversity of a set of choices. Towards the end of the paper they observe that some of their recommender systems increase the experience of diversity for every individual in the sample and yet decrease the overall diversity of the culture. So I wrote a program that does basically what they do in their paper and tweaked it to highlight this result.

The result is what’s important here, rather than the particular algorithm used to generate this instance of it. But I know some people will want to know how the results are generated, so I’ll give a short sketch. If you want more than this, Fleder and Hosanagar provide details, my tweaks to their model are available as source code (python) if you want, and if you post in the comments we could get into a discussion. But it’s not important, trust me.

Each simulation starts with 48 customers and 48 products. Each product is described by two attributes, with values generated according to a normal distribution. So the products are distributed on a two-dimensional grid, with a value of about -3 to +3 along each axis. Each customer is assigned a taste for each attribute, so they also are scattered about in the same space. The idea is that a customer will prefer, other things being equal, a product that is close to it in these attributes. Here are two distributions of customers (blue) and products (red). You can see that most customers share a mainstream taste around the middle of the graph, but there are a few who have odd tastes off to the edges. Likewise, most products have attributes that are mainstream, but there are a few “niche” products closer to the edge.

In this particular simulation, a customer can choose the same item over and over again, so it simulates something like streaming radio more than a bookstore. Each simulation starts off with a priming phase, in which each customer makes 75 choices according to a function which favours nearby products, but with some randomness so that they may on occasion choose one further away. After 75 choices we turn on a recommender function. Whenever a customer goes to make a choice, the recommender system identifies a product and recommends it to the customer. The recommendation increases the chance that the customer will choose the recommended product. Fleder and Hosanagar look at a few recommender functions. The one I use works like this:

  • The set of 48 customers is divided into equal-sized communities, with members chosen at random so they may not be close in taste.
  • The recommender function chooses an item by looking at what customers in the same community have chosen. It recommends the one most popular among others in the community.

I’m just going to show you two simulations. Run 1 above – which I will call Internet World – treats the entire set of 48 customers as a single community. The other (run 28 above), which I will call Offline World, breaks it into 24 communities of two people each. In Offline World I will get recommendations from the people around me and you will get recommendations from the people around you, but these recommendations are separate and isolated. In Internet World we each get recommendations from all 48 customers.

Here are the results for the two simulation runs I’m going to focus on. The results of these simulations are far from the only possible outcome, but they show why the gut feeling may fail, and I’ve chosen them for that purpose.

In Internet World each customer experiences an average of 3.5 products over the course of 75 choices with an active recommender system, while in Offline World each customer experiences only 2.4 different products. So the wider set of people providing recommendations in Internet World has led to an increase in individual diversity. This is like saying that “Netflix shows me pictures I would never had heard about from my friends alone”, or “Amazon recommended a book I had never heard of, and I liked it”.

On the other hand, the overall diversity of the culture can be measured by the Gini coefficient of the products. A Gini coefficient of zero is complete equality (each product is chosen an equal number of times) and a Gini coefficient of 1 is complete inequality (only one product is ever chosen by anyone). And Internet World has a Gini of 0.79 while Offline World has a Gini of only 0.52. Internet World is less diverse than Offline World.

How can these seemingly contradictory results happen? Let’s take a look.

In the following graph, each dot is a customer, arranged in their two-attribute preference space (just like in the graphs above). But this time the area of each dot is proportional to the number of unique products they experience. So in Run 1 (Internet World) you can see that the dots are, on average, bigger than the dots in Run 28 (Offline World). This shows the greater individual experience of diversity in Internet World; for example, there is a customer with attributes of (1.1, -0.8) who samples no less than 38 different products, and only seven of the 48 customers stay with a single product throughout the whole simulation. Meanwhile in Offline World  the most eclectic customer samples only nine and there are no fewer than 19 customers who sample just one product. The experience of individual customers in Internet World is of broader horizons and more selection, as recommendations pour in from far and wide, rather than from the limited experiences of their small community in Offline World. This picture has become the standard narrative of choice in the Internet World – our cultural experiences, liberated from the parochial tastes and limited awareness of those who happen to live close to us, are broadened by exposure to the wisdom of crowds, and the result is variety, diversity, and democratization. It is the age of the niche.


But wait!

Here is a graph of the products in each simulation. This time, the area of each dot shows its popularity: how often a customer chooses it.

You can see that on the left, in Internet World, a few products were chosen a lot, especially the one centred on about (-0.2, -0.2). In Offline World there are many more medium-sized dots, showing that the consumption of products is more equal. In Internet World one product has “gone viral” and gets chosen over 1500 times out of the total of 3600, while 26 products languish in the obscurity of being sampled fewer than ten times. In Offline World no single product is chosen more than 10% of the time, and only 14 products are sampled fewer than ten times. In short, niche products do better in Offline World than in Internet World.

While each customer on average experiences more unique products in Internet World, the recommender system generates a correlation among the customers. To use a geographical analogy, in Internet World the customers see further, but they are all looking out from the same tall hilltop. In Offline World individual customers are standing on different, lower, hilltops. They may not see as far individually, but more of the ground is visible to someone. In Internet World, a lot of the ground cannot be seen by anyone because they are all standing on the same big hilltop.

The end result is the Gini values mentioned before. Here are Lorentz curves for Internet World (blue) and Offline World (green), in which the products are lined up in order of increasing popularity along the x axis, and the cumulative choices for those products is plotted up the Y axis. 

So there it is. Individual diversity and cultural homogeneity coexisting in what we might call monopoly populism.

But don’t think this is just about automated recommender systems, like the ones that Amazon and Netflix use. The recommender “system” could be anything that tends to build on its own popularity, including word of mouth. A couple of weeks ago someone pointed me to this video of Madin, a six-year-old soccer prodigy from Algeria, and the next day my son, who moves in very different online circles to me, was watching the same one. I know who Jim Cramer is even though we don’t get CNBC in Canada because everyone is talking about him and helping his disembodied head to shoot down Jon Stewart. More people watched Tina Fey being Sarah Palin online than on Saturday Night Live, and Fey is now famous in countries where no one watches the TV show. Clay Shirky writes an essay and I get five different links to it in my Google Reader feed in one morning. Our online experiences are heavily correlated, and we end up with monopoly populism.

A “niche”, remember, is a protected and hidden recess or cranny, not just another row in a big database. Ecological niches need protection from the surrounding harsh environment if they are to thrive. Simply putting lots of music into a single online iTunes store is no recipe for a broad, niche-friendly culture.

Mr. Amazon’s Bookshop: The French Lieutenant’s Bookshop?

  By

[This would be the thirteenth episode of Mr. Amazon's Bookshop, if it were an episode. The previous episode is here. A list of all episodes is here. In the previous episode, Kylie and Edmund vanished from Whimsley Hall after finding out that Mr. Amazon's recommendations were doing nothing to help sell Kylie's story The Adventures of Wazzock. Whimsley sunk into a brief depression before being roused by Jennie the one-legged housekeeper, who sent him into the village just in time to see Kylie and a pack of teenagers heading towards Mr. Amazon's Bookshop with trouble on their minds.]

I wrote the first ten episodes of this story in a couple of weeks around Christmas, when I had some time off work and the story seemed fresh and interesting. The two most recent episodes have been written on weekends in and around cooking, cleaning and so on. And now we have all the principals with the exception of Jennie the one-legged housekeeper – Whimsley, Google, Kylie, Edmund (who is in the crowd, near the back, though you might not have noticed him), Mr. Amazon -  in one place, and something dramatic is obviously about to happen. But what?

Now I don't want to go all John Fowles here (see the book in the title) and pontificate about the nature of the plot as artifice and the role of the author. That's so last century. Plus, when Fowles wrote that he could send his heroine this way or that, and when he wrote his alternative endings, he had actually finished his book, so the questioning of his role as author was a little precious.

My problem is different. I don't have two ways to finish the story, I have none. In fact, there is no story from here on.

I have some ideas about things that might happen next, including what seems to me a rather neat twist about the obviously impending assault on Mr. Amazon's Bookshop, but they are vague and they have problems. Not least among them is my growing tiredness with Whimsley himself. His one-dimensional nature was a benefit in Mr. Google's Guidebook (written just over a year ago! how time flies), but after stretching him over 15,000 words rather than 2,000 I am not sure there is much more to be said about the internet from the point of view of a delusional alcoholic gothic anachronism.

I did think Kylie may provide an alternative focus for the story, but she has fallen somewhere between a youthful village mob-boss, a smart and witty kid, and a yokel, without one aspect of her character really crystallizing in my imagination.

And then there is the technology issue. Clearly the Kindle has to come in to the plot somehow. Having Google already on scene means that a last-minute twist driven by the Google Book Settlement might be a nice way to finish off the tale. Perhaps Kylie drives Amazon out of town only to hand Google her stories for him to distribute? And I am sure that after all this it will turn out that there are indeed mole-people, but that they are working for Google, not Amazon, slavishly copying all those books for his ever-expanding guidebook.

So we will see. I realize this raises questions about the responsibility of the author to finish a story once started. I would like to finish it, and I expect that I will do so. But there will be no episode this week.

This is not a fish for attention and reinforcement, by the way. I have appreciated the comments on earlier episodes. And to prove that it is not, I will close comments on this posting. It's just an observation that I have other things going on, like work and family and friends and books and spring, and I have no episodes stocked away ready to be brought out and posted. If anyone wishes to continue the story on their own blog, feel free: I will be happy to link to it.

So with that, see you next week, with luck.

Mr. Amazon’s Bookshop: Where is Kylie?

  By

[This is the twelfth episode of Mr. Amazon’s Bookshop. The previous episode is here. A list of all episodes is here. In the previous episode, Kylie and Edmund spent a morning running Whimsley’s differ and made some disturbing findings about the Mr. Amazon’s recommendations. Then Kylie left without warning to go and steal some lunch, leaving Whimsley even more bemused than usual.]

After the excitement and confusion of Kylie and Edmund’s investigation, evening found me tired and emotional. My head was spinning with all the numbers they had collected and what Kylie seemed to make of them. You may not believe this, but I was feeling a little out of my depth. At sea even. We gentry typically have little use for numbers bigger than a dozen or so; anything more and we have hired help to deal with them. And though my interest in the differ had got me more familiar with numerology than most of my class, I had always been more enthusiastic about building the device than actually using it.

I had written Kylie off as riff-raff when I first met her, but I was beginning to realize that some of these youngsters may actually know a thing or two. Her outburst about Mr. Amazon’s unfairness carried the mark of immaturity and self-dramatization so characteristic of her class and sex, but as I pored over the notebook I really did not know what to make of the reams of Edmund’s writing in my notebook. Yesterday Kylie had showed me one page of summary and claimed that is showed Mr. Amazon’s populist claims to be fraudulent, but as I turned page after page of my notebook I could make neither head nor tail of the admirably neat tables and lists that I saw in front of me. My first thought was that it would be embarrassing to have to ask for more detailed explanations, but then I realized that numerology is really a matter for the trades, not for men of standing, so there is no shame in being a little foggy when it comes to details. My neighbour Mr. Belloc was correct when he told me of his friend’s fate:

Lord Finchley tried to mend the electric light
It struck him dead, and serve him right.
It is the duty of the wealthy man
To give employment to the artisan.

By then I was looking forward to Kylie’s return, when I would demand further explanation of her investigations and take our next steps in what seemed increasingly like a campaign against Mr. Amazon and his book “shop”. I had the bit between my teeth and was ready to run – I felt energized and zealous.

But Kylie did not reappear the next day. I fretted, and paced the grounds.

And she did not appear the following day either. I went by the gardener’s hut to ask Edmund her whereabouts, but Edmund had vanished as well! I expressed my concern to his father, who told me Edmund frequently went to stay with relatives in the village and that I “should not worry my little head”, which seemed charming enough, but was hardly helpful.

When neither was to be seen on the third morning, my spirits tumbled and I retreated into the gloom of Whimsley Hall. Their explosion of youthful energy reminded me what a grim dwelling it has become over the years, and their sudden absence plunged me into weeks of sour temper. The stone pillars, the sepulchural halls, the long shadows – they speak to my bones through the generations of Whimsleys, and yet there are times when I resent their weight and their lugubrious depths. A darkness came over me in those days, despite the summer sunshine outside. I spent morning after morning in the breakfast room just staring at the patterns made by the damp on the ornate wallpaper, afternoons in the library poring slack-jawed over erotic tales and cheap pulp novels. The evenings in fitful, chemically-assisted slumber.

Google would bring me the newspaper each morning and tell me the odd facts he always has at his fingertips that would normally keep me amused, yet they could do nothing to lift my spirits. Jennie would clatter into the room, swinging on her crutch, to bring me my breakfast or to clean away a few of the every-accumulating cobwebs. Usually a friendly sight, she became a Barquentine-like presence, stunted and cantakerous and clouded with a deep rage. And why not? He family has served mine for six generations, tied almost as intimately as myself to the arches and history of Whimsley Hall. She has watched from close up my failure to maintain its legacy, to prevent its fall. The world was changing around me, I felt. Whimsley Hall has been, truth to tell, an anachronism for years. And now, in the new world of the prim Mr. Amazon and his impersonally-friendly recommendations, what place would there be for the traditional virtues of short sharp floggings, of loyalty and doffings of caps and tied cottages? Ritual and symbol were fading. I could see no way to raise myself from despondency.

It was Jennie who dragged me out of my self-absorpion, as autumn approached and the winds and rain came back to scour Whimsley village. One morning she hobbled in with my usual breakfast tray of fried eggs, fried bacon, fried black pudding, fried potatoes, fried sausages, fried mushrooms, fried tomatoes and fried bread, together with a half-full bowl of soup. She was breathing deeply, clearly exhausted from her journey up the staircase as she cleared a spot on the desk and threw the tray unceremoniously in front of me. “There you go. Full English. Not that it will help.”

“What do you mean, help?” Jennie rarely spoke these days, so I was taken aback.

“You’ve not been eating right for weeks. Ever since that excitement with Kylie and Edmund and the differ. You’re going downhill in a big way sir, and I’m not going to stand by and watch it happen. You need to pull yourself together.”

Of course, if there is one thing guaranteed not to help a person in spiritual pain it is telling them to pull themselves together. I scowled and stabbed a fork half-hearedly at a half-hearted sausage.

“You’re fretting about them numbers Kylie and Edmund put together ain’t you? She’s too quick for the likes of you, that one.”

“You know Kylie?”

“Well”, she said in a most un-servant-like tone, “they don’t call it Whimsley village for nothing. It’s not exactly surprising if I live down the street from Kylie Higgins is it? There’s not that many streets. And even if I didn’t, everyone knows Kylie. You can’t miss that one, what with running the bookie’s for her Dad. She’s smart, that kid, and she puts up with no nonsense.”

“The bookie’s eh? That would explain her facility with numbers. Do you know where she is? She said she would be back, but then she and Edmund just vanished.”

“Well she’s been up to no good, I can tell you that. Sometimes I sit on my porch of an evening you know. Watch the dogs fight, just to pass the time. It’s not easy to get around a lot, what with having one leg. Not that some people care, getting me to hop up and down staircases all bloody day long.”

“A little exercise will do you good, I’m sure. Just you wait and see. But what do you mean ‘up to no good’?”

“What I heard was that she decided Mr. Amazon’s a fraud. So she’s been rounding up some of the local kids – teenagers with nowt to do over the summer holidays. She started off by telling them stories about this Wazzock character and his dragon and they loved it. But then last night she just stopped. They got mad and she just says ‘Well if you want to know more you’d better go and get Mr. Bloody Amazon to sell you a book then.’ They didn’t look so keen on that, so she says ‘And if you don’t want to buy one, maybe you could just take it. Who’s to stop you?'” So I reckon she’s got it in for this Amazon and she’s going to set the neighbourhood kids on him – break in and do the place over,
if you get my meaning. Maybe they’ve already done it.”

“I’ve no doubt she’s capable of vandalism. But vandalism won’t help sell her book, will it? I think I’ll have to talk some sense into the young harridan. We pillars of the community have a duty, don’t you know?”

“You betcha. You’d better not waste any time if you want to stop her. You’ll probably find her at the bookie’s or in the park by the pond. So you won’t be needing these eggs then? Not to worry, I’ll look after them.” And with that Jennie whisked the tray from under my nose, and clattered off towards the staircase before I could say anything, spilling the remaining soup in all directions as she went.

It was indeed time, I decided, to take action. Raising myself from my torpor I flung open the windows and breathed in the stale odour of the Whimsley air. I found my walking stick, put on my heaviest boots and within a few minutes I was striding off down the road to the village, with the cobwebs of my mind left among the cobwebs of Whimsley Hall. Just as I approached the gate I noticed Google talking to the gardener. He’s a big man and might be handy if there’s trouble, I decided. “You’re with me Google!” I cried, “There’s trouble at Mr. Amazon’s shop and we need to be the voice of reason. I know you don’t like the man – not too keen on him myself – but we can’t have hooliganism in Whimsley village. Not cricket is it?” I’m gratified to say that Google looked shocked. I can tell you that doesn’t happen often. He dropped his conversation like a hot potato and came after me, and the gardener followed too.

We approached the village and turned up the street on which Mr. Amazon’s bookshop was found, and were greeted by the sight of a crowd of youth – perhaps twenty of them – approaching the shop from the other end. They looked tense and nervous; sleeves rolled up, ready for business. Kylie was at their head, but she didn’t even acknowledge my presence. Instead the crowd went straight to the bookshop and she yanked at the door.

It did not open.

“Hey, Amazon!” she yelled. “I want some answers. And I want you to sell some copies of my book. I’ve got some customers here, so let us in, right?” She hammered on the door again.

It looked like there was going to be trouble. How delicious!

Mr. Amazon’s Bookshop: Recommending the Big Sellers

  By

[This is the eleventh episode of Mr. Amazon’s Bookshop. A list of all episodes is here; the previous episode is here. In the previous episode Kylie began to think that Mr. Amazon might be “just like the bastard publishers”, pushing big sellers at customers rather than promoting books like her own novel The Adventures of Wazzock. Meanwhile, Whimsley was becoming increasingly befuddled by the swirl of activity around him. His befuddlement even interfered with his sleep….]

I had trouble sleeping that night. It wasn’t just the heat mixing with the cat-urine-induced damp of the carpets to produce the acidic tang that so characterizes Whimsley Hall in the summer; I find that odour reminds me of my own childhood and is surprisingly comforting. No, Kylie’s virulence had quite upset me and my usual tonics did not seem to relax me as they usually do. But finally I slept fitfully, only to dream…

I looked out of my bedroom window to see a book lying in the middle of the vegetable garden. As I watched, the book opened and a vine grew from its spine, each page becoming a leaf. The vine sprouted pods periodically along its length, and each one erupted to vomit a new book onto the ground. These new books sprouted vines in turn, and the vegetable garden was soon a mass of twisting green creepers, writhing ever closer to the walls of Whimsley Hall. Before I knew what was happening, they were appearing at the window, and each leaf bore the face of Mr. Amazon. “If you like vegetables”, said one, “maybe you would be interested in this,” and the leaf rotted on the stalk, emitting a stench of rotten turnip. “If you like books”, said another “maybe you would like a phonograph, or some toys, or some gourmet groceries”, and an avalanche of bread, mechanical devices, soup, toy soldiers, and other contrivances flooded the floor of my bedroom. “It’s a new economy of abundance!” crowed a face to my left; “It’s a world of choice!” cheered one to my right. More and more, faster and faster, the faces wriggled and wormed their way over the windowsill, in through the ceiling, up through the floor, until I could hardly move

I woke suddenly, tangled in the bedclothes, sweating and shaking. The room was empty, the window closed, the night quiet. I squirmed out from the sheets, ran to the window and stared out, but everything was peaceful. Reassured but exhausted, I put my nightcap back on my head and returned to my bed, and drifted back to slumber. But just as I was about to lose consciousness, I thought I heard Mr. Amazon’s voice again. “If you want a vision of the future, imagine a human trying on a new boot – forever.”

The following day dawned grey and wet. The rain battered the windows, and I felt a mixture of relief and disappointment as I realized Kylie was unlikely to venture all the way from the village to Whimsley Hall today, but before I had even started in on my kippers there was a knock at the window.

“What’re you doing Mr. W? Me and the imbecile had been busy for hours. Come and have a look.”

I ran to the door, and sure enough, steam was billowing from the stable chimney. The differ was running. I stood straight, called for an umbrella, and set out to join the urchins. Jennie the one-legged housekeeper carried the umbrella for me so that I could finish my kippers and tea on the way over. The umbrella kept knocking against my head as she bounced up and down. It really is most inconvenient to keep her on sometimes, but one does what one can.

“So tell me,” I demanded as I strode – impressively I’m sure – into the stables.  “What have you been doing with my differ?”

“Well,” started Kylie, “the undersized lamebrain here was watching you when you set up your dials to send the questions to Mr. Amazon. So we worked out how to send our own questions. And we chose another book, and we grew a shop from it. We did the same as you did: choose one book as the seed; visit about a thousand times. Each visit, we look at summat between 0 and 20 books, and the shortarse wrote down all the output.” She pointed at Edmund, asleep in the corner. “This time we chose something called The Shack by William Paul Young. It’s a one of a kind invitation to journey to the very heart of God, and a lot of people seem to like it. So we’ve got the whole bookshop down in your notebook. You’ll be needing some new cats by the way.” She gestured to a damp pile of fur on the floor opposite the dials.

“And what did you find?”

“In short, a lot. Here, why don’t I pour you a drink and you can take a seat while I tell you all about it.” She poured what looked like a stiff gin into a cup and handed it over.

I took the drink with thanks, and sat attentively.

“First things first. In the Shack shop, we spent half the time looking at books in the top 2204 sellers, which is pretty much just the same as the Special Topics in Calamity Physics book shop. And four fifths of the time is spent in the top 22,700, which is five thousand less than yesterday’s bookshop, and there’s only one in ten views that are outside the top one hundred grand. So that’s slanted even more to the best sellers than the Special Topics shop.”

“You’ve obviously been busy,” I was having a hard time following all these numbers, but Kylie ploughed on remorselessly.

“So dummkopf here, he’s pumping the bellows like mad and writing it all down and he says ‘Let’s do some more Kylie!’, so we did. He’s quite the bundle of energy this kid”, She punched him amiably in the stomach and Edmund stirred in his sleep. “I’ll show you the results here.”

A sheet of paper in my notebook had a summary of each of the shops they had created, the median sales rank, and the time spent in browsing around the top 100,000 sellers. Here it is.
Store Median  Top 100K   Seed Book
 7     1742    87.5    Testimony
 4     2007    91.4    The Lucky One
 9     2113    89.4    The Shock Doctrine
10     2113    87.6    Special Topics in Calamity Physics
12     2191    86.5    Our Mutual Friend
 1     2192    89.8    The Shack
29     2238    98.3    Bones
20     2276    88.2    The Shock Doctrine
22     2388    88.5    The Long Tail
17     2471    92.5    Our Mutual Friend
 8     2650    86.6    Conservatism That Can Win Again
21     2808    90.4    The Conscience of a Liberal
24     2888    90.1    The Long Tail
27     3135    88.8    The Long Tail
 5     3158    96.0    Seven Habits of Highly Successful People
19     3380    89.1    The Long Tail
 6     3438    91.2    No One Makes You Shop at Wal-Mart
14     3487    88.2    The Conscience of a Liberal
25     3610    88.3    The Long Tail
23     3627    93.3    The Long Tail
26     3681    91.5    The Long Tail
18     4060    84.8    The Conscience of a Liberal
13     4101    88.0    Surrender is Not an Option
16     5326    90.0    Riddley Walker
30     5750    87.9    Graph Drawing
28     6117    88.3    The Secret History
 2     6222    86.2    How To Eat
15     7231    84.6    Our Mutual Friend
31     7583    100     The Secret History
 2    17794    76.3    Last Best Gifts
11    51992    58.5    The Dilbert Principle

I pored over the numbers, unsure what to make of them. “So explain, snotnose” I grumbled. Somehow this venture did not feel entirely mine any more.

“It’s like this,” the vixen explained, slightly patronizingly I felt, “We did a number of books as seeds to see how things went differently depending on your starting point.”

“But some have the same book name.”

“That’s because each bookshop may go different ways, depending on what choices the cat tells you to make, and we wanted to see if the results depended a lot on what book you start with. So would you always get the same shop if you start with one book.”

“And do you?”

“Sometimes yes, sometimes no. Like, we did Our Mutual Friend twice because we both did Charles Dickens at school and hated it. Can’t that man say anything in less than twenty pages? And you can see that one of them ended up with a median of 2471 and one with a median of over 7,000. That’s quite a difference on this list. And we did The Long Tail quite a few times because Mr. Amazon told me about that when I asked him how he could help The Adventures of Wazzock to get what it deserves. But in the end, with a few exceptions it doesn’t make much difference what book you start with. You’ll spend at least half your time in the top 10,000 books, which is like browsing around the heavy traffic area of a regular bookshop. And you’ll spend almost all your time in the top 100,000, so you don’t really see much that you wouldn’t see in a normal bookshop.”

“What about these exceptions you mention?”

“There are three that are different, and all for the same reason. Look at these bottom three in the list. The Secret History, Last Best Gifts, and The Dilbert Principle.”

“What about them?”

“Well The Dilbert Principle is the book that spends most of its time out in the outsider part of the shop, beyond the 100 grand mark. Any idea why?”

“Well I have heard that Dilbert is an humorous illustration of some kind. Isn’t that right?”

“Yeah, so what you get is, you bounce around a lot of other comic books. Every one of the top 14 books is a Dilbert book and from what I can tell all but two of the top 80 are comics. There’s not really any one big bestseller in comics, so that’s why you spend so much time in the outsider books.”

“And what about Last Best Gifts?”

“A bit different. It’s basically an academic book and you end up going round a lot of other academic books. You don’t always do that – like Graph Theory is pretty much like a bestseller – but Last Best Gifts seems to just wander the academic ghetto. What a bunch of losers!”

“And The Secret History?”

“Now that one is very strange.” Kylie frowned and scratched her stomach, contemplating the oddities of the world. “This shop has the least number of books of all. Ten thousand different views and there’s only 14 different books in the lot! The reason is that early on it gets into books by this bloke Christopher Moore who, it says here, writes offbeat humour. And then all the recommendations are for other Christopher Moore books. And the really odd thing is that none of those books appear in any of the other bookshops. You either like Christopher Moore or you don’t I guess.”

“So what have we learned?”

“Well, first that it’s really easy to get distracted by these little stories of what makes one bookshop different from another.”

“And second?”

“That’ll have to wait until tomorrow. I’m hungry and I’m going to steal some lunch.” And with that she stalked, hands deep in pockets, down the drive towards the village.

The Brilliant Bechdel/Wallace Movie Test

  By

I had not heard of this amazing test until seeing a mention in the morning paper, but it has been around since 1985. That was when Liz Wallace told it to her friend Alison Bechdel who put it in the comic strip Dykes to Watch Out For. See here for the comic and here for discussion.

For others who have been in the dark, here is their devastatingly simple rule.

To be worth watching, a movie must
  1. Have at least two women in it,
  2. who talk to each other, 
  3. about something besides a man. 

This morning's paper went through the Time Magazine top ten movies of all time list. Classics like The Godfather and 2001 A Space Odyssey fail. I have a feeling both TV programs I watched this evening fail. And I'm half way through reading The Unbearable Lightness of Being, which I think is likely to fail as well.

Review: The Future of the Internet and How To Stop It by Jonathan Zittrain

  By
The New York Times recently asked: Do We Need a New Internet?:
At Stanford, where the software protocols for original Internet were designed, researchers are creating a system to make it possible to slide a more advanced network quietly underneath today’s Internet. By the end of the summer it will be running on eight campus networks around the country.

The idea is to build a new Internet with improved security and the capabilities to support a new generation of not-yet-invented Internet applications, as well as to do some things the current Internet does poorly — such as supporting mobile users.

The Stanford Clean Slate project won’t by itself solve all the main security issues of the Internet, but it will equip software and hardware designers with a toolkit to make security features a more integral part of the network and ultimately give law enforcement officials more effective ways of tracking criminals through cyberspace.

Ed Felten of Princeton University responds with an orthodox hacker-purist line:

[The first misconception] is the notion that today's security problems are caused by weaknesses in the network itself. In fact, the vast majority of our problems occur on, and are caused by weaknesses in, the endpoint devices: computers, mobile phones, and other widgets that connect to the Net. The problem is not that the Net is broken or malfunctioning, it's that the endpoint devices are misbehaving — so the best solution is to secure the endpoint devices…
It's an appeal to ye-olde Internet mythologie, complete with deferential references to "the founders" and their foresight, as if the Internet were some real-world Seldonian Foundation.

Neither position is good enough, and Jonathan Zittrain's wise book The Future of the Internet And How to Stop It [home page, Open Library entry] does a great job of explaining why and of providing some better ways of thinking about the problem. Yes, designing security into the Internet will inevitably cripple the very flexibility and permissiveness that has made the Internet a continuing source of unpredictable and surprising innovations. But sentimental idealization of the Original Internet and its "end-to-end" design won't do either [p165].

[U]sers are not well-positioned to painstakingly maintain their machines against attack, leading them to prefer locked down PCs [as Felten appears to advocate, ed], which carry far worse, if different, problems. Those who favor end-to-end principles … should realize that intentional inaction at the network level may be self-defeating, because consumers may demand locked-down endpoint environments that promise security and stability with minimum user upkeep. This is a problem for the power user and consumer alike.

…When endpoints are locked down, and producers are unable to deliver innovative products directly to users, openness in the middle of the network becomes meaningless. Open highways do not mean freedom when they are so dangerous that one never ventures from the house.

The passage shows the strengths and weaknesses of the book. Zittrain takes an unusually and refreshingly pragmatic, realistic view of the Internet, rejecting old approaches as and when needed. On the other hand, you can see from the first two sentences that his prose can be repetitious and dry. You have to work your way through some dense thickets to get to through this forest – but if you are ready to make the effort, it's a worthwhile journey: one of the best Internet books I've read.

Zittrain's central concern is a dialectical contradiction at the heart the Internet:

  1. The Internet's success comes from its remarkable ability to repeatedly generate new and unexpected uses.
  2. This "generativity" (yuck, what a clumsy word), comes in turn from the deliberate dumbness of the Internet protocols themselves; they were deliberately designed to permit any kind of traffic, for any purpose, to pass between end points of the network.
  3. But the more the Internet becomes "prime time", the more it attracts spammers, virus writers and information thieves to prey on the online population, and the Internet's dumbness is free for them to use as well.
  4. The more these highway predators threaten our online experience, the more we seek to retreat to the safety of a closed and protected world.
  5. A closed and protected world may give us safety, but will spell the end of the Internet as a fount of innovation.

The Internet, like capitalism, contains the seeds of its own destruction. But Zittrain is a digital reformer, not a revolutionary. Inspired by the continuing success of Wikipedia in the face of similar problems, he favours a combination of light regulation (like health and safety standards for the Net) and popular community action (neighbourhood watch). He believes that these, combined, can preserve the creative spirit that has led to so many innovations, while staving off the worst of the security and other problems. And he makes a solid case, buttressed by broad research (50 pages of notes and references) and a careful, undogmatic and pragmatic attitude.

The book is in three parts. Part I is a history of the Internet, told to highlight two design principles that were present right in the original TCP/IP architecture. The first is the "procrastination principle", which says the "the network itself should not be designed to do anything that can be taken care of by its users" [p31]. Most features of a network should be implemented at its computer endpoints (the end-to-end principle) rather than "in the middle". The second principle is trust; the Internet is "a bucket-brigade partnership in which network neighbors pass along each other's packets", and the assumption of co-operation and fair dealing is present in its design. So the Internet has no built-in security or identification mechanism; anyone can join the network; and there is no quality of service guarantee for packets it delivers. These two principles have led to what Zittrain calls the "generative dilemma" [36]. "The idea of a Net-wide set of ethics has evaporated as the network has become so ubiquitous" [45]. So how do we regain security while maintaining the ability to be creative?

Part II outlines what Zittrain sees as some of the dangers facing the Internet. One danger is the rise of Internet appliances, such as many of today's mobile phones, X-boxes, and Kindles. These devices promise a secure environment, but at the cost of restricting the ability of programmers to be creative. The second is at the other end of the network, where "Web 2.0" platforms such as Google Maps, Facebook, Salesforce and other hosted environments offer "contingent" environments for programming, where the prospect of unilateral changes to terms of service or agreements inhibits creativity.

Zittrain is enthusiastic about much of what the Internet has wrought, but he parts company from current digital orthodoxy on these issues. When it comes to Web 2.0, influential commentators from Shirky to Lessig to Tapscott and even to Benkler gloss over the differences between commercial web sites and non-profits, in an attempt to highlight a unified Internet culture. Lessig, for example, talks optimistically of a hybrid economy in which these motivations muddle along next to each other, and while Benkler does see an opposition of interests between the market economy and the collaborative network economy, he never makes much of it. I haven't seen as clear-minded an analysis of why commercial Web 2.0 platforms threaten creativity as Zittrain provides and I agree with him wholeheartedly.

The mobile Internet is only beginning to get the kind of attention that Web 2.0 has received. Today's smartphones and yesterday's desktop computers are similar in terms of computational power (see this claim of Windows 3.1 running on a Nokia N95 if you don't believe me). But whereas Microsoft got hauled in front of the courts to keep the desktop computing environment open for non-Microsoft applications and opposing the bundling of Internet Explorer, there are no such worries for mobile phone vendors as they keep their proprietary app stores and their managed devices in the name of security. There are layers of the iPhone, the BlackBerry – yes, and Android devices too – that are open only to the operating system, and which third-party applications cannot access.

A natural response to Zittrain's worry that we face an Internet of closed appliances (or, as Margo Seltzer calls them, "gizmos") and closed services is that these can exist in parallel with open, "generative" devices and communications. But Zittrain rejects this particular compromise:

even in a world of locked-down PCs there will remain old-fashioned generative PCs for professional technical audiences to use. But this view is too narrow. We ought to see the possibilities and benefits of PC generativity made available to everyone, including the millions of people who give no thought to future uses when they obtain PCs, and end up delighted at the new uses to which they can put their machines. And without this ready market, those professional developers would have far more obstacles to reaching critical mass with their creations.[165]

The end-to-end principle, argues Zittrain, has had its day, as "'middle' and 'endpoint' are no longer subtle enough to capture the important emerging features of the Internet/PC landscape" [167].  In its stead he advocates a more general principle that seeks explicitly to maintain "generativity", so that ISP filtering of viruses may be worth considering, for example.

Zittrain sees an inspiration in Wikipedia, whose shambolic, after-the-fact, bits-and-pieces way of fixing problems has been one of its strengths. Zittrain gives two initiatives he has been involved with, that could drive a similar approach for security while maintaining generativity. One is herdict, a browser plugin that collects the input of people from around the world to assess web site accessibility. Another is stopbadware.org , which helps to maintain a list of web sites that may host viruses and other badware. These initiatives extend the now commonplace collection of user experience by commercial software providers and service providers by making the collected data universally available. The distinction between, on the one hand, Google's use of our searches and Microsoft's collection of Windows crash reports and, on the other hand, herdict's goal of making the collected data public is a big one. It shines a light both on the closed nature of the major corporate collectors (no matter how unevil they may claim to be) and on the possibilities of open data. Public access to current databases of virus reports, spam outbreaks, and so on provides a basis for innovative solutions to the worst damages inflicted by malware.

Maintaining a healthy digital environment – both security and generative – is a commons problem, and while dictatorship is an appealing route to take to solve the security half of the issue, real transparency provides a second possibility that has more chance of solving both. But it needs to be real transparency, not just the half-hearted efforts of Web 2.0 companies as they dance along the line between community and profit.

There's a lot more in the book. On the technological front Zittrain sees virtualization technologies as reason for optimism and I agree; I'm getting a new computer at work next week and I anticipate running everything in virtual machines. If I get a virus or my registry gets screwed up, I'll roll back to yesterday's state and it will be gone. Well, that's the plan. Let's see how it works. On the social and political side, the book's final pages tend to wander aroun
d issues of privacy, reputation, and behavioural norms without saying a whole lot that's new. It's a shame the book finishes on this topic, because it fizzles out a bit. But don't let that deter you – there is a lot of subtle insight and a real breadth of knowledge in these pages that you rarely see in books about the cutting edge of technology, and Zittrain's efforts deserve to generate a big discussion.

Mr. Amazon’s Bookshop: Another Conversation with Google

  By

[This is the tenth episode of Mr. Amazon’s Bookshop. There will probably be another five or six before I’m done. A list of all episodes is here; the previous episode is here.]

I rang for Google who, as so often, appeared quicker than seemed reasonable, with his usual false air of subservience. 

“How the hell do you get here so quickly Google?” I demanded, slightly taken aback by the speed with which he materialized at my shoulder. 

“Oh, nothing special sir – I just keep a copy of myself close by in case you need anything. It may seem a little unusual, but it’s entirely neutral. Any other butler could do the same. Certainly no one is leveraging their unilateral control over your bell to hamper user choice, competition, and innovation.”

I often regret asking too many questions of Google. I never know when he is being funny and when he’s not. Usually I decide to treat his remarks as humour because life is just easier that way, but he has an excellent poker face so it’s hard to tell.

“Well, never mind. Listen, we have some questions for you. It’s about Mr. Amazon.”

Google’s face showed an instant distaste: “I’ll do what I can.”

I was just searching for the best way to phrase my questions when Kylie piped up.

“Hey Mr. G. We’re trying to find out whether Mr. Amazon’s recommendations are going to help me to sell lots of copies of The Adventures of Wazzock. We need to know some things about Mr. Amazon’s sales. Like how much he sells that you couldn’t get in a regular establishment bookshop.” She spat at Edmund as she said this last sentence.

“A fine question young miss. Let’s see. I am ashamed to say I know very little about Mr. Amazon’s sales in any detail. I have means to find out many things, but he seems capable of great secrecy. But from what I hear it is possible that he makes about 25% of his sales from books beyond the 100,000 mark. Such measurement depends greatly, of course, on what length of time the sales ranks are averaged over. A book that is positioned at 150,000 could easily put itself into the top 100,000 by a single sale – so do you count the sale as before or after the purchase? Still, I imagine the figure is not too far from the truth.”

“When it comes to real bookshops, a small independent bookshop may stock 30,000 titles. But of course they sell special orders too. Perhaps ten percent of their sales are special orders. A large chain store like Heather’s Big House O’Books may stock 100,000 titles. But there are larger ones still. I am told that Blackwells in Oxford, for example, stocks above 200,000 distinct titles in its shops on The Broad.”

“You’re a gem Mr. G. That’s what we need to know.”

“It is?” I asked.

“Obviously”, said Kylie. “Think a bit. He makes one sale in every four from outside the top 100 grand books. But when he mutters his ‘would you be interested’ guff, we look at these outsider books only about once in eight times. So he’s pushing the top sellers, innit? He’s only selling them others because he’s a convenient way to order books you already know about, which is probably just taking business from the regular bookshops where you might have ordered them before. And quit the boozing, we’ve got work to do.”

It took a moment before I realized she was talking to me. I sheepishly replaced the cap on the flask of cognac and returned it to my overcoat pocket.

“Hey shortbus,” Kylie was approaching Edmund, who flinched. “Let’s have a look at that other graph of yours. What’s it show?

Later on that day I managed to get a look at the graph, which I reproduce below. Meanwhile, Kylie was telling us all about it.

“So if I read this right, you’ve got your views of a book up the side, and your sales rank along the bottom. And most of the books Mr. Amazon shows us are bunched up at the left among the mainstream establishment junk, which we knew already. But it also shows that the ones he shows many times lot are almost all best sellers. Look at them buggers that have come up more than 50 times! Let’s see” (she scoured the notebook again, brow furiously furrowed.) “There’s 31 books that we looked at over 50 times. Twenty three of those 31 are in the top grand. All but two of them is in the top two grand. And I bet not one of them is The Adventures of Wazzock. He’s just showing everyone the establishment books innit? I thought he was all friendly to us kids, but it looks like he’s just like the bastard publishers! I should have known. He’s all about the money. What an absolute pillock.” She spat on the ground as she said this.

Such was her virulence I confess that I actually felt sorry for Mr. Amazon, who is after all just trying to make a living.

“Let’s not leap to conclusions young Kylie. After all, this is just one sample, like you said. Maybe if we set up the differ again and start with another one, we’ll see something different. And would you like an ice cream?”

Her expression remained ominously grim.

“I’m not being bought off with no ice cream. But you’re right Mr. W. We’ll have to try again. I’ll be back tomorrow morning. You be ready or else, numskull.” She kicked Edmund affectionately in the shins as she stomped off back down the drive.