Category: reputation-systems

So why the trouble with Yelp? Well, according to Cerretini, Yelp has been continually pestering the restaurant to advertise on the web site “30 times a week, sometimes five times a day”. This may be an exaggeration. Still, for six months, Botto Bistro did advertise. Then they cancelled their ads but Yelp kept calling. “I insulted them and I called them bad names”, says Cerretini, “Yelp asked us to pay them as advertising so they can criticize me better.”

In frustration, the Botto Bistro owners flung up their hands (metaphorically, I wouldn’t want to stereotype Italians) and started their Hate us on Yelp campaign to become the worst-reviewed restaurant in the San Francisco area. Thirteen hundred reviews later, they proudly announced a one-star aggregate rating on Yelp, the lowest rating possible.¹

Their Yelp page is a lot of fun. Some reviewers compete for the most outrageous complaint (“Their pizza ruined my marriage”, “I’m sitting here in my office in Hamburg/Germany and are in need of some food and learn that you are not able to deliver???”, “WORST place EVER!” and so on). Others are more obviously supportive (“Had this same idea when Yelp screwed over my business… just never went about doing it! Very glad to see it working out for you.” “Fuck you, Yelp. Long live Botto.”) Everyone is having a good time.

But Yelp has, stupidly enough, responded, as Faravar tells us. On September 15 “Marvin” from Yelp sent one of the co-owners an email that started this way:

Hi Davide,

I ‘m contacting you from the Yelp User Support Team because we’ve received complaints from the community that you may be offering incentives in exchange for reviews.

To be clear, this violates our Terms of Service…

After accusing Davide of violating Terms of Service to which he has not agreed, the email goes on to threaten “a Consumer Alert being placed on your listing”.

So three cheers for Botto Bistro. Its campaign is many things: it’s a gimmick, a gesture of frustration, a declaration of individuality and quirkiness, and an act of rebellion against the creeping tyranny of a world defined by software and Terms of Service.

But the campaign relies on novelty for its success, so it’s not a tactic that can be easily reproduced by others. What, then, can other restaurants do if they are frustrated by Yelp? It’s clear that some resort to “fake reviews”, with less panache than Botto Bistro. It’s an act that’s widely seen as both ethically and legally wrong, but maybe there is a place for it…

We all know what’s wrong with fake reviews. Crowd-sourced sites such as Amazon’s book reviews, TripAdvisor’s hotel ratings, and Yelp’s restaurant recommendations, depend for their success on being the honest and authentic opinion of those who have read, stayed at, or eaten at the subject of their review. If restaurant owners reward people, or pay people, to submit positive reviews (sometimes people who have not even eaten at the restaurant) then those reviews are “frauds”: Yelp talks about “fake” or “phony” reviews, and calls them “attempts to deceive consumers”.

Some law enforcement agencies agree that paying for reviews qualifies legally as fraud: last year New York regulators carried out a sting operation that got “19 companies to cease their misleading practices and pay a total of $350,000 in penalties.”

“What we’ve found is even worse than old-fashioned false advertising,” said Eric T. Schneiderman, the New York attorney general. “When you look at a billboard, you can tell it’s a paid advertisement – but on Yelp or Citysearch, you assume you’re reading authentic consumer opinions, making this practice even more deceiving.”

The finding of “false advertising” was not based on whether the content of the reviews matched reality, but on the fact that the reviewers were posing as authentic customers, while actually being paid. But of course there is nothing unusual in a business paying somebody to speak positively about their offering. We’ve all seen television advertisements featuring celebrities who are paid large amounts of money to endorse a product or service.

The claim that it’s worse when it’s not on an obvious billboard is also outdated. We all know that TV and movie product placements are endorsements inserted precisely to seem natural – where you can’t “tell it’s a paid advertisement” – by actors who may never have used the product they are endorsing. We’ve heard about “viral marketing” stunts in which people are paid to act as genuine enthusiasts for a product. This may be a bit sleazy, but it is also just business as usual.

There are many places where advertisement and “authentic” comment is getting mixed up. It’s of real concern where the venue is a public venue: the practice of drug companies promoting their own brand of painkillers in university lectures is an obvious one, but I’m sure you can think of more.

But why, at a time when ads and “authentic” comment mix in Facebook, Twitter and YouTube, is the Yelp reputation system off-limits as an advertising medium? Why is it “false” to pay people to say nice things about your business on Yelp’s review listings but not “false” to solicit endorsements on LinkedIn (tip: my LinkedIn profile is almost entirely fake), in the middle of a prime time television show? Why should Yelp reviews not be just another venue for marketing and promotional activities?

It’s well known that Section 230 of the Communications Decency Act protects web site owners from being responsible for user-provided content on their sites, even when that content is the source of profits. The protection extends to Yelp, according to a judge in 2011 and reaffirmed in September 2014. Yelp has no legal responsibility to verify any comments made by reviewers, for factual accuracy, reasonableness, or anything else. So if a restaurant complains that a reviewer’s opinion is not based on fact, that’s not Yelp’s problem. Opinions can be both “authentic” and wrong.

But of course Yelp does not just assemble user content, it also arranges it and presents it. Yelp’s sorting algorithms can make a big difference to a restaurant’s business: is the company bound to be fair in its presentation of restaurants? Business owners recently took Yelp to court claiming that the company “extorted advertising payments from them by manipulating reviews on its site”. A US Federal Appeals Court found that—whether or not Yelp is manipulating reviews—it can do what it wants.

…even if small business owners could prove that Yelp manipulates their pages to persuade them to pay up for advertising, that doesn’t mean they are being extorted, or that Yelp is violating federal law… the law does not require Yelp to publish positive reviews in the first place, Judge Berzon wrote.

“By withholding the benefit of these positive reviews,” the judge wrote, “Yelp is withholding a benefit that Yelp makes possible and maintains. It has no obligation to do so, however.”

So Yelp is fundamentally unaccountable for the fairness of its reputation system.

The usual argument is that it is in Yelp’s own interest to maintain a trustworthy site—that market forces will keep it honest. But while it’s clear that a completely spam-filled site would lose its value, it is not clear that the optimal amount of “review fraud” from Yelp’s point of view is zero. Yelp is interested in volume as well as accuracy: it encourages participation by reviewers, restaurants, and users through a whole gamut of techniques. There is always going to be a balance between this encouragement and dissuasion of so-called “biased” reviewers. We cannot trust loosely-defined market forces to preserve fairness.

Sometimes Yelp foots the bill directly for the venue, food or drink; oftentimes we look for sponsors to help defray those costs (e.g., Peroni Beer comped the beer I drank last week in New York). The venues themselves typically provide the space, some staff and sometimes food or drink. This arrangement is great for yelpers because we can visit a new place and meet each other affordably. And it works well for the venues and sponsors because they get exposure to a group of local adventurers who, if they like what they see, may become future customers.

There’s a clear quid-pro-quo here: free stuff for reviews. Not that it’s explicit, but of course it doesn’t have to be. Ironically, Yelp understands how the implicit quid-pro-quo works when it comes to the small businesses being reviewed: here is an excerpt from Yelp’s advice pages for businesses:

Why does Yelp discourage businesses from asking for reviews?

Would-be customers might not trust you. Let’s face it, most business owners are only going to ask for reviews from their happy customers, not the unhappy ones. Over time, these self-selected reviews create bias in the business listing – a bias that savvy consumers can smell from a mile away. No business is perfect, and it’s impossible to please 100% of your customers 100% of the time.

Business owners obviously feel the implicit promise when approached by Yelp. According to the Chicago Tribune, back in 2009 Chicago cupcake shop owner Patty Rothman was approached by a Yelp staffer who “walked into her Gold Coast shop and ‘guaranteed us good reviews on the site if we catered one of their parties for free’. Offended but resigned, Rothman complied. And just as promised, positive reviews bloomed for the business right after the party, Rothman said.”

Or Doug Quint and Bryan Petroff of Big Gay Ice Cream in New York, which opened in 2011:

As Quint and Petroff explain, last summer before their New York City shop even opened its doors for the first time, they got an email from a so-called “Elite” Yelper that read, “I am going to bring a team of other Elite Yelpers on an ice cream crawl and we want to come to your store. Can you open early?” And then, in addition to asking the Big Gay Ice Cream guys to come up with special flavors just for them and sample-size portion of the whole menu, this Elite Yelper went in for the kill: “Think of all the great reviews you’ll get.” (link)

From the LA Times in August 2013, reporting on a town hall for business owners:

…when the town hall opened up for Q&A an hour and a half later, business owners were quick to vent their frustrations.

Many slammed the company for allowing review posters to write inflammatory comments – one restaurant manager said she cried for three days after a Yelper said her restaurant was filled with Nazis, another said he’d received negative reviews from Yelpers who admitted they’d never tried the food – while others said they had been subjected to aggressive advertising calls from Yelp.

“I have one-star reviews for my diner from people that have never walked into the place. They’ve never stepped foot, they’ve never tried the food, but they give me one-star reviews. That’s insane. Why would you let someone like that stay on the site?” demanded Craig Martin, owner of Cafe 50s. “I spoke to your office, I called your guys, I emailed, I talked to your salespeople.”

Another business owner, vintage clothing shop owner Reiko Roberts, said the advertising pressure amounted to extortion. She said that when she declined to buy ads, “the lower reviews go to the top and the higher reviews go to the bottom.”

From a Reddit Ask Me Anything session with the CEO, here are two of many:

My anecdotal experience: I told the Yelp sales rep to get lost and to stop calling me after about 6 months of hounding. He finally relented… but guess what ALL of my positive reviews are now filtered. Up until that conversation, only about 25% of my reviews were filtered. Now EVERY SINGLE ONE.

I sat in on a sales call with a Yelp rep, and after questioning her on the finer details we decided Yelp wasn’t for us and told her so. She then became noticeably irritated and said “Good luck getting any positive reviews in the future.”

And for more, you can look (where else), on Yelp’s page on Yelp, where it has a rating of 2.5 and 3105 one-star reviews.

You might wonder if these claims are true. Is it fair of me to post these allegations without checking into their accuracy?

Well, one reason restaurant owners get angry at Yelp is that Yelp does not care whether factual claims made by reviews are true or verifiable: their rankings of businesses are based on opinion, regardless of veracity.

So I’m happy to tell you that my sophisticated, but unfortunately proprietary, algorithms have identified all these reports of Yelp as honest opinions of the people involved. I’m sure you can find Yelp’s response to these allegations elsewhere, but they haven’t created an account here on Whimsley, so I’m not going to post a link. What’s sauce for the goose is sauce for the gander.

While the net effects of reputation systems are complex and varied, there is a homogenizing, winner-take-all component that comes from this practice of ranking preferences and displaying them in order of average value. Yelp heavily favours those businesses that appear in the top few of the listings.

Yelp’s decision to rank based on average reviews favours restaurants with a broad appeal, and acts against more polarizing venues. It’s the same reason that many challenging or controversial movies that divide audiences (“cult movies” belong in this class) score lower on Rotten Tomatoes than those that are less divisive. Cult movies may attract a more intense following among their core audience, but there is nowhere in the Rotten Tomatoes system for such intensity to be reflected.

Like a cult movie, Botto Bistro’s owners set out to deliver their own vision in an uncompromising fashion. Appeasing those who don’t like it is not high on their list of priorities. As they told Faravar: “We specifically, aggressively inform people that if you’re looking for customer service, this is the last place you should be.”

There is a place for many types of restaurant, including those with Botto’s take-it-or-leave-it style. But it’s clear why Botto Bistro will never get far on Yelp even if they wanted to: Yelp’s whole model of reputation works against the idiosyncratic and difficult.

The homogenizing force of “average and rank” systems may become more severe as whole industries seek to increase the role of digital ratings in our daily lives. There are many in the technology world who dream of reputation as currency: the idea that a numerical reputation score on one system (such as Yelp) should be portable to other environments (TripAdvisor? Credit Card companies? Landlords?) A good score on eBay may indicate a “good reputation” that you could parlay into a better interest rate on a bank loan. It’s already happening to some extent, as Airbnb posts the number of Facebook Friends of each host as evidence of their trustworthiness. This kind of naive scoring system is a terrible idea. The only thing worse would be a sophisticated scoring system.

The aggregation process itself can also affect your business. A couple of years ago Rebecca Rosen reported in The Atlantic on a study by Michael Anderson and Jeremy Magruder: the researchers seized on the observation that two restaurants can have very similar average ratings (say 3.24 and 3.26) and yet when Yelp rounds them into half-star categories one will be rounded down to 3.0, while the other is rounded up to 3.5. Looking at 400 restaurants in the San Francisco area, they discovered that “moving from 3 to 3.5 stars is associated with being 21 percentage points more likely to have sold out all 7:00 PM tables and moving from 3.5 to 4 stars makes restaurants an additional 19 percentage points more likely to have sold out all tables.” That is, Yelp’s own choices about how to present ratings, independent of business “quality” reflected in its average rating had a significant effect on some businesses.

In short, the impact of almost any reputation system on a restaurant’s business is not just the natural effect of that business’s quality, but is also a result of the system designer’s choices, which cannot be influenced by the business owner. Business owners are dependent, to some extent, on the design choices of an unaccountable company for their success or failure. That’s not a pleasant position to be in and it’s made worse by the “winner-take-all” structure of many social media marketplaces. If a few fake reviews are needed to limit the impact of large reputation systems, that’s fine with me.

Let’s finish up with what philosophers call “the question of meaning.” If we are to believe in a “fair reputation system” we have to ask: what, grasshopper, does a one-star (or a five-star) review really mean?

Yelp’s view (and TripAdvisor’s, and Netflix’s, and Amazon’s, and Airbnb’s, and eBay’s … you get the picture) is that a rating is an indication of individual preference, and that aggregating these preferences can be used to recommend the experience to others. If “reputation” is the social distillation of others’ opinions about ourselves, then these rankings are a measure of “reputation”, distilled multiple times into a single number.

But just because that’s what the aggregators would like a rating to mean doesn’t make it so. There is a world of context behind each click that may not fit into any algorithm’s assumptions. In Botto Bistro’s case, a one-star review is a gesture of support, a way to thumb your nose at an impersonal system, and a quick path to cheap pizza. It’s a dramatic example of how one “one-star review” may have a completely different meaning from another “one-star review” even though they are recorded in Yelp’s systems in the same way.

It’s tempting to think that if only “fake” reviews could be removed, then the remaining ones would indicate preference, but there’s little to back up such a claim. In other contexts (like eBay or Airbnb) a five-star review is more often a courtesy than a rating. Less dramatically, in the case of Netflix, one outcome of the Netflix prize was that reviews of one movie are heavily primed by what other movies we have recently reviewed (so that “objectivity” is tainted).

And maybe a “fake review” should be seen as a commitment on the part of the restaurant owner, an investment in branding that indicates an intent to stay in the marketplace, a sign that the restaurant can, in fact, be trusted — just like any other investment in branding.

Ratings do not always mean what data scientists would like them to mean. Human expressions are protean, and no attempt by Yelp or others to pin them down will succeed completely. There is something postmodern about ratings, just as any other human expression. We imbue them with meaning and intent outside of what the system wants, and there is no objective fact behind it all that is the “true” rating. Il n’y a pas de hors-click.

Reputation is a multi-faceted, qualitative concept. It has been pushed through a meat-grinder by digital reputation systems and has come out the other side homogenized, devoid of texture, but easier to digest. There’s nothing inherently wrong with reputation systems by themselves, but giving them too much authority or influence will inevitably throw up bad incentives on the part of the system owner and those taking part in the reviews. Before we declare “fake reviews” to be a crime or an obvious case of bad morals, we at least need to demand some accountability from reputation site owners.