The shape of Airbnb’s business (II)

Collecting data for several cities both in November 2013 and in May 2014 gives a look at some ways in which Airbnb’s business is changing in these cities. The cities surveyed are:

  • Paris and New York (Airbnb’s two biggest markets)
  • San Francisco (the home of Airbnb)
  • London, Rome, Berlin (three European capitals and major tourist cities)
  • Toronto and Chicago (two smaller markets).

As in Part I, the surveys were carried out by a two-stage method:

  • Stage 1 used the Airbnb search pages for a city to collect Room ID values (a number that uniquely identifies an individual listing).
  • Stage 2 visited the listing page for each Room ID and collected information about it.

The data were stored in a database and the charts here are based on queries of that database. The survey code (and database) are available on github.

The main observations are:

  • It is possible that the number of listings has reached a maximum in some of Airbnb’s largest cities, particularly in the USA, while other cities do continue to show significant growth.
  • There is a surprising amount of churn in Airbnb’s business. Over a third of the listings on the site in November were no longer there in May, to be replaced by a similar (or, in some cases, greater) number of new listings.
  • Around 90% of all ratings on the site are 4.5 stars or 5 stars (out of five).

These observations suggest some conclusions:

  • Airbnb’s move to professionalize its host base may be a reaction to the churn in the host population, a necessary step to generate continued growth.
  • In April Airbnb expelled 2,000 listings from its New York offerings, claiming that they did not offer a positive experience. Many of those listings must have had either no ratings at all or were rated highly (4.5 or 5) by their guests.
  • The rating system does not provide a reliable assessment of host quality.

Churn

Figure 1 shows the net change in the number of listings in each of the eight cities for which data was collected in both November and May, as a percentage of its November value.

Some cities seem to have maxed out in terms of listings. Six of the cities saw either decreases or increases of less than 10% in the total number of listings. Each of the three US cities in the study lost listings between November and May, while Berlin and Rome were big gainers.

nysuffer.png

Figure 1: Change in number of listings in cities.

The net gains or losses mask a bigger change. Figure 2 shows that in the six month period, well over a third of existing listings vanished from the site, and were replaced by a similar (or, in the case of Berlin and Rome, substantially greater) number of new listings. For both Paris and New York, well over 40% of listings vanished, only to be replaced by new ones. Figure 3 shows the total number of vanished and new listings.

The high churn rate must introduce uncertainty for Airbnb’s future, as it is relying on a steady stream of new listings to replace those that are vanishing from the site. Perhaps this is one of the motivators of Airbnb’s efforts to professionalize its host base: hosts who make a commitment to the business being more likely to stick around.

churn.png

Figure 2: New and vanished listings between November 2013 and May 2014, as a percentage of the November number.

example1.png

Figure 3: absolute number of new and vanished listings

[Note: I did wonder whether this high churn rate could be an artefact: whether the displayed room ID values could have changed between the November and May data collections. A query showed that, of the 10597 listings that occur in both the November and May collections from New York, 10529 have the same host. I concluded that room ID values are stable.]

We can look a little closer at those vanished listings. Figure 4 shows that the vanishing listings have fewer reviews than the overall population in the city.

churn_reviewcount.png

Figure 4: Mean number of reviews for vanished listings and for all listings

Figure 5 shows the ratings that the vanished listings had received from guests. The overall distribution of ratings is investigated later in this report, but Figure 5 shows that, while many of the listings had no reviews at all (Airbnb recently claimed that reviews are left for 70% of visits, so it is likely that these listings had no visits either), of those that did have visits the average rating was overwhelmingly high (4.5–5).

churn_lost_ratings.png

Figure 5: The number of vanished listings by average overall rating

Figure 6 emphasizes the point, showing that the average rating of listings leaving the site is within 0.1 of those that stay on the site.

churn_rating.png

Figure 6: Average rating of listings that leave the site (red) and which have been on the site for the whole period (green). Note that the y axis starts at 4.5.

It seems likely that many hosts try Airbnb, have a few guests (or none at all) and then simply decide that, for whatever reason, the service is not for them. To repeat the conclusion from the overall churn rate, the high turnover rate does raise questions about what long-term population of hosts the site can support, and how it can ensure that those who try and leave the site are replaced by new hosts.

Trust

Trust has long been a key part of the Airbnb business, and technology has been seen as the magic ingredient. Airbnb CEO Brian Chesky expresses the company’s confidence when he says of city-level rules that “they’re primarily set up for screening. To protect consumers. Well it turns out that cities can’t screen as well as technologies can screen. Companies have these magical things called reputation systems… We think government should exist as the place of last recourse.”

There are two sides to Airbnb’s trust system. There is peer-to-peer trust, often described as a reputation system, which consists of ratings, personal profiles, and comments written by hosts or guests and seen by other hosts or guests. The other side is centralized trust, which consists of central payment, verified ID, and a complaint system.

Much of the talk around the sharing economy focuses on reputation and peer-to-peer trust (as in Chesky’s comment above). That’s because it’s the peer-to-peer aspect that is new: after all, in one sense we have forever trusted strangers to deliver services from food to haircuts to regular taxis, and the centralized component of the system is not that different to many hospitality companies. Jason Tanz writes in Wired that

We are entrusting complete strangers with our most valuable possessions, our personal experiences—and our very lives. In the process, we are entering a new era of Internet-enabled intimacy.

So what do the numbers have to say about peer-to-peer trust?

Figure 7 shows the distribution of ratings for each listing. Airbnb provides ratings under a number of categories (cleanliness, accuracy of description etc) and collects them together as “overall satisfaction”. The site does not give the ratings for individuals (although individual comments are visible) but provides an aggregate, from 0 to 5 stars with a half-star granularity.

city_rating_distribution.png

Figure 7: Overall satisfaction ratings for listings in each city. The ratings for 0-3 are so few that they are collected together. Listings with no ratings are excluded.

The vast majority of ratings are either 4.5 or 5 stars. The finding is consistent with ratings on other sites (e.g. an earlier look at BlaBlaCar, or more serious studies of eBay listed there). When we rate each other, ratings become more a courtesy than a judgment. Just as restaurant tips only weakly correspond to the quality of service, so a rating of 4.5 or 5 is more a way of politely concluding an exchange than it is of assessing the behaviour of a host or guest.

The ratings obviously fail to distinguish among the people on the site, and so are not providing the service for which they were intended. This is part of the reason why Airbnb and others have moved to emphasize the centralized trust aspect of their systems. But centralized trust is the same reason we trust the fast-food restaurant cook or the hotel cleaning staff. The major sharing economy sites rely on discipline and the potential for removal from the site to provide security for the users, just as traditional industries do.

Churn and Trust in New York

The Airbnb court case in New York raised the profile of the service there, and resulted in Airbnb ejecting 2,000 listings from its service.

Airbnb claims that “we found that some property managers weren’t providing a quality, local experience to guests. These hosts weren’t making their neighborhood stronger and they weren’t delivering the kind of hospitality our guests expect and deserve. In some cases, they were making communities worse, not better.”

With 10,000 listings vanishing from Airbnb in the New York area, it is difficult to know which 2,000 were removed by Airbnb and which left for other reasons. Some individual hosts with large numbers of listings have been removed, but that still leaves many unaccounted for. Figure ny-vanished shows that, despite Airbnb’s suggestion that the hosts are not “delivering the kind of hospitality our guests expect”, most of the removed listings must have had either no ratings at all, or must have had good ratings from their guests.

ny-vanished.png

Figure 8: The “overall satisfaction” ratings for listings in New York that vanished from the site between November 2013 and May 2014.

The biggest single category of the 10,000 New York listings that have vanished from the Airbnb site are listings with no ratings at all (shown as NULL). After that come the 5.0 and 4.5 ratings, leaving fewer than 800 with bad ratings (4 or lower). For at least 1200 of the listings that Airbnb removed, bad ratings is not the reason.

Combined with the very high number of positive ratings shown in Figure 7, this result shows that the significance of peer-to-peer ratings is exaggerated for Airbnb. While the company proclaims that peer-to-peer rating systems set it apart from old-style methods, it is clear that their trust system is essentially a traditional complaint-based, centralized system. The peer-to-peer ratings may give warm feelings on the site, but Airbnb itself clearly does not trust it when it goes to remove listings from its site.

Conclusions

The data set suggests that there is a surprising amount of churn in Airbnb’s hosts. Over a third of the host population in November left the site, to be replaced by new listings.

It is likely that a large number of people experiment with being Airbnb hosts, and then decide that it is not for them.

It will be interesting to see whether this becomes an issue for Airbnb as the number of ratings it has in the North American cities surveyed in November and May seems to have peaked. How will it continue the growth that is demanded of it by its backers in the face of the large number of hosts who drop off the platform?

While much is made of the novel peer-to-peer nature of Airbnb’s reputation system, it appears that the company runs a complaint-based trust system that is not so different from other hospitality companies. In April Airbnb expelled 2,000 listings from its New York offerings, claiming that they did not offer a positive experience, but many of those listings had either no ratings at all or were rated highly (4.5 or 5) by their guests: it appears that Airbnb did not use its own ratings when judging who to remove from its platform.

Bookmark the permalink.

6 Comments

  1. Great work! I think this kind of documentary evidence is vital and I wanted to take a moment to acknowledge your huge contribution to a real understanding of how these services really work.

    I remember a while back you wrote honestly about the complications of balancing this kind of work with the “paying the mortgage” side of life. I hope you’ve found a good solution, because you’re doing something important here.

  2. Tom, I don’t find the churn nor trust numbers as surprising as you do. Churn can be accounted for by owners using short-term Airbnb listings to fill the gaps between longer term rental leases, owners switching between intermediaries that handle the listings on their behalf, as well as owners testing the short-term rental market and not liking what they find as you indicate (there is probably other important cases too).

    I think the ratings of short-term rentals is different than ratings for movies/restaurants/hotels. I’m guessing that their power is in gauging the accuracy of the listing rather than as a comparison tool between properties.

    Complaints vs. Ranking/Reviews is interesting. My initial thought was that complaints should somehow be reflected in the public review system but I can imagine cases in which an individual would prefer to keep their complaints private. I wonder if there is a parallel between rental-ranking/rental-complaint and app-ranking/bug-reporting. I’m guessing this is a simple UI issue rather than an inherent flaw in peer-to-peer reputation systems for short-term rentals.

    • On the surprise: I’m sure there were a variety of reasons. I was just surprised to see about half of the existing listings drop off over a period of six months. But apparently that’s just me.

      I really think the ratings have very little informational quality – not even about the accuracy of the listing. The eBay and other cases suggest it really is a courtesy. Like tips, or notes in guest books when leaving a place — it has to be pretty bad before most people say something negative. If 30% don’t leave a review, then it’s likely that potential complaints are kept silent.

      The reciprocity factor is part of this, but I don’t think it’s all. I think the fact that it’s a person you are reviewing (and one who can review you) makes it different to app-ranking rather than a UI issue. For one thing, the prevalence of five-star reviews exists on most person-to-person ratings (eBay, Blablacar, Airbnb, Lyft, Uber).

  3. The prevalence of the five-star ratings could be due to the “J Curve”

    J Curve

    • I hadn’t seen that, so thanks for the link. The ratings here are even more weighted to the high end than the product ratings shown there (it’s a J with a very long vertical line) and I think the stats above show that the “bad” listings are not being removed from the system over time, but it’s an interesting thought anyway.

  4. This is excellent work Tom.
    I did a much smaller study in the Melbourne suburb of Richmond a few months ago. About 100 listings.

    I was astonished at the high growth rate in Airbnb listings over the past 12 months, of which most was in the most recent 6 months. That is, that local market was going through its first growth cycle, a few years behind the San Francisco/ NY experience.
    When I sampled the availability calendars, it seemed that many owners were listing just so they could rent when they were away on holidays for 2-4 weeks of the year. For many of these, I can imagine a high churn rate as they question being publicly listed all year for a few weeks rental.

    I also found that the Airbnb model had 8 layers of trust, which makes it very attractive for owners and renters alike.
    As users find the model attractive for rentals of all kinds not just spare city rooms, I expect that its penetration will extend from inner city to more traditional vacation rental areas in the regions far from the city.

    This is likely to fuel continuing exponential growth for a year or 2, even as early adopter cities hit saturation and churn matches growth as you point out.
    The Airbnb global exponential growth has been truly astonishing, and listings in Airbnb are poised to soar past traditional rental websites like Homeaway in a few short years.
    We are for an interesting time as we watch this market disrupter change (or not change) the rental industry.

Comments are closed