Developing a Culture of Experimentation at Airbnb

---

Controlled experiments should be used at every step of the product development process to learn and make decisions. Experimentation is equally important in shaping product decisions and user experiences.

---

You may also enjoy these articles:

• Pricing The Silicon Valley Way - Van Westendorp Model

• Strengthen Your Strategy With Experimentation

• Be Hypothesis Driven, Not Idea Led

• Overview of Business Experimentation

• Sequence Experiments to Decrease Risk and Uncertainty

Introduction

Airbnb is one of the world’s most successful companies. The company has a relentless obsession on the customer, designing amazing user experiences for guests and hosts alike.

From the outset, Airbnb has had a strong focus on data-informed decision-making and experimentation.

The path to scaled experimentation at Airbnb was a journey over many years. The business now executes hundreds and hundreds of experiments every week.

Building a culture of experimentation at Airbnb was supported by three core elements - building platforms and tools to support experimentation, embedding experimentation into key business functions and actively shaping a culture of experimentation by hiring data-minded people.

We all have dreams of scaling up experimentation. Whether your business is just starting out with experimentation, or running hundreds of experiments every week, there’s some lessons to be learned from Airbnb.

Overview

When Airbnb first started out with experimentation, they were designing and executing less than 100 experiments per week. Now, the business is running more than 700+ experiments per week.

Such a significant uplift in business experimentation does not happen overnight. It comes from fostering and cultivating a data-informed culture over many years.

Key to successfully scaling experimentation at AirBnB was:

• Building dedicated tools and platforms to support experimentation

• Embedding experimentation into key business functions

• Actively shaping a robust culture of experimentation

Experimentation is hard work. However, it does get easier over time.

“We used controlled experiments to learn and make decisions at every step of product development, from design to algorithms. They are equally important in shaping the user experience.” – Jan Overgoor

For experimentation to work in your business it must come from a place of natural curiosity, with a really strong desire to learn from your customers.

Airbnb experimenting with the Host sign up page user experience …

Experimentation at Airbnb

Some of the key elements for developing a culture of experimentation at Airbnb were:

1. Invest in a data-minded team

2. Data isn’t numbers, it’s people

3. Embedding data scientists into teams

4. Experimentation should be hypothesis-led, not feature driven

5. Aligning around a measurable north star metric

6. Building platforms and tools to support experimentation

1. Invest in data-minded people

One of the early hires that Airbnb made was a data scientist, Riley Newman. Riley was hire number 10.

At the time, nobody had even heard of Airbnb, let alone could pronounce the name. Airbnb was still operating out of the apartment of founders Joe Gebbia, Nathan Blecharczyk and Brian Chesky.

Making the appointment of a data scientist in the early stages of the company lifecycle was a strong signal. Data was going to be central to help the team learn and evolve as fast as possible.

“Bringing me on was a forward-looking move on the part of our founders. This was just prior to the big data craze and the conventional wisdom that data can be a defining competitive advantage.

Back then, it was a lot more common to build a data team later in a company’s lifecycle. But they were eager to learn and evolve as fast as possible, and I was attracted to the company’s culture and mission.

We knew so little about the business that any insight was groundbreaking. The company was so small that everyone was in the loop about every decision; and the data team (me) was aligned around a singular set of metrics and methodologies”

– Riley Newman

2. Data isn’t numbers, its people

Data is at its most useful when it can be trenched and disseminated through a business. This is how a data-driven culture can flourish naturally and organically.

How a business thinks about, and uses data, is essential to embedding data-driven thinking inside business functions.

Where data becomes most powerful is when it is translated from cold, hard facts into the “voice of the customer”.

Data from customer journeys can be used to identify purchasing preferences and behaviours.

This information is super useful to inform business decision-making, prioritisation, resource allocation and product development.

“Data science is an act of interpretation – we translate the customers voice into a language more suitable for decision-making.

This idea resonates at Airbnb because listening to guests and hosts is core to our culture. Since the early days, our team has met with community members to understand how to make our product better suit their needs. We still do this, but the scale of the community is now beyond the point where it’s feasible to connect with everyone everywhere.

Data has become an ally. We use statistics to understand individual experiences and aggregate those experiences to identify trends across the community; those trends inform decisions about where to drive the business”.

- Riley Newman

3. Embedding data scientists into teams

Often, data teams can be viewed as a transactional interaction.

I have a question:

• How many nights booked in New York?

• How many listings are there in Paris?

The data team can provide information to answer my question. Hey presto.

The realisation that the data science team at AirBnB offered far more value than answering ad-hoc questions and measuring things led to changes in structuring.

There was a shift in the data science operating model from centralised, to decentralised.

Data scientists were allocated to customer journeys to solve specific business problems, partnering closely with Product Managers and Engineers in dedicated, cross-functional teams.

This model has served Airbnb well and is still in effect today.

Decentralising any business function can come with downside, primarily through a loss of quality control.

“Experimentation needs to be embraced and evangelised throughout the org. One thing that made this quite easy at Airbnb was the fact that data scientists were embedded in product teams. Data scientists were not just 'consulted' on product, but data itself became critical to decision-making.

Data scientists sat next to PM’s and along with the engineers and designers were a critical voice in roadmap planning. I find the strongest teams operate with data at the outset – they don't just call over the data scientist when it's time to do a post-facto analysis of how XYZ experiment went. Some of the best product ideas we came up with on Airbnb's early growth team came from data scientists.

At one point the Growth team started bi-weekly experiment reviews where product team members would present on experiments they ran, with a focus on lessons learned, mistakes and next steps. Knowledge sharing was off the charts.”

– Lindsay Pettingill

4. Experimentation should be hypothesis-led, not feature driven

Experimentation is founded on the scientific method. This approach is an iterative learning model that is hypothesis-led.

A good hypothesis should be grounded in some sort of reality, existing facts, observations or information. A hypothesis is not a guess, it’s a theory about what may work in the future.

A hypothesis should be based off existing quantitative or qualitative information.

While it’s possible to build almost anything these days, just because we can build a new feature or product, it doesn’t mean that we should.

If you’re guessing, you should stop and do more research before you start experimenting.

“Why do hypotheses matter? Without them you’re untethered, easily distracted by what appear to be positive results, but could well be statistical flukes.

 In this situation it’s easy to make up a story that fits your findings as opposed to doing the hard work of understanding what’s going on. Things may surprise you — let them!”

– Lindsay Pettingill

5. Aligning around a measurable North Star metric

One of the most impactful shifts in AirBnB’s experimentation culture came from aligning around a single quantitative north-star metric: nights booked.

Before this, there was no shared understanding of what teams should optimize for and how experiments could inform those decisions:

“As the company grew, hunger for analysis overwhelmed my team. At the time I chose what we worked on based on what would be most impactful to the business.

But how do you compare growth in China against machine learning improvements for our search ranking algorithm?

The CFO was experiencing a similar version of this problem, so we came up with the idea of establishing a north star metric: nights booked. We would use this as a common denominator for projects so we could align on where to prioritise investment.

An added (and unexpected) benefit of doing this was it propelled the company's use of experiments.

Leaders were suddenly held accountable to measurable results that weren't as obvious for how to hit relative to simply completing tasks. They needed to know why some things worked and others didn't, so they began embracing experiments and analysis. It changed the whole dynamic of our interaction.”

— Riley Newman

6. Building platforms and tools to support experimentation

To make experimentation easier in the business, an Experiment Reporting Framework (ERF) was built.

When designing this tool, the key focus was to make experiments simple to run. This was achieved by reducing the chance of errors or biases with experiments and automating analytics.

Airbnb Experimentation Reporting Framework (ERF)

There were four key design principles for the platform:

1. Make sure the underlying data is correct

2. Limit the ways someone setting up an experiment could accidentally introduce bias

3. Experiment review processes were the same as BAU reviews

4. Automate experiment analysis to keep the barriers to running experiments as low as possible

“While the basic principles behind controlled experiments are relatively straightforward, using experiments in a complex online ecosystem like Airbnb during fast-paced product development can lead to a number of common pitfalls.

Some, like stopping an experiment too soon, are relevant to most experiments. Others, like the issue of introducing bias on a marketplace level, start becoming relevant for a more specialized application like Airbnb”.

- Jan Overgoor

Conclusion

Running controlled experiments is a great way to inform decisions around product development and user experience.

The lessons from a high-performing experimentation culture at Airbnb provide unique insight into key pinch points and focus areas for your business.

Establishing experimentation at scale is not easy. It takes a lot of time and effort.

Sustained organisational commitment is required over many years to cultivate and nurture a strong culture of experimentation.

For experimentation to succeed, it requires data-minded resources to be hired, organisational structures aligned to support and drive experimentation, and investment in tools and platforms to make experimentation easy.

Data should be your ally. Identifying trends and patterns with your product and customers is a great way to drive your business forward.

---

Need help with your next experiment?

Whether you’ve never run an experiment before, or you’ve run hundreds, I’m passionate about coaching people to run more effective experiments.

Are you struggling with experimentation in any way?

Let’s talk, and I’ll help you.

---

References:

Riley Newman, Lindsay Pettingill, Jan Overmoor