Lukas Vermeer - A Global Leader In Online Experimentation
“There is no such thing as the perfect experiment. The notion of a perfect experiment doesn’t exist. There is only a continuum between very well executed experiments, and completely winging it off the seat of your pants. The intent is not to make a few high-quality decisions and guess the rest, but to make all decisions a little bit better”
You might also enjoy these articles:
Six Lessons From The Award Winning bol.com Experimentation Program
United Nations: Solving The World’s Toughest Challenges With Experimentation
Introduction
Lukas Vermeer is widely acknowledged as a global leader in the field of online experimentation.
He is an expert at establishing and scaling experimentation and is currently the Director of Experimentation at Vistaprint, a global Dutch eCommerce company that produces physical and digital marketing products for small businesses. It has more than 5,100 employees and generated revenues of more than $3B in 2022.
Lukas has co-authored many influential academic papers on experimentation and is a highly sought-after conference speaker.
He is an experimentation practitioner at heart, combining industry experience in A/B Testing and Data Science with a background in computing science and machine learning.
Lukas is best known for his work as Director of Experimentation at Booking.com, the world’s leading accommodation website.
At Booking.com he was responsible for the tools, infrastructure, and training that helped product teams at Booking.com improve the customer experience through hundreds of thousands of experiments.
In this article we discuss:
Optimising the experimentation flywheel
Identify threats to validity
Making more decisions a little bit better
Experimentation as product management
Establish Communities of Practice to scale experimentation fast
1. Optimising the experimentation flywheel
Removing friction and bottlenecks
Every company that runs experiments wants to perform more.
“If you want to scale experimentation in your organisation, really, the only thing that you need to do is, identify where the friction is, and then remove it. That’s the single thing that you need to be obsessing about. What is preventing you from performing twice as many experiments? What is the bottleneck now?”
And then you need to pull the thread a little and see whether you can remove it, circumvent it, or grease the wheel a little bit.
If you have a bottleneck in your experimentation process, and you optimize just before the bottleneck, you make things worse, not better. The work will pile up in front of the bottleneck.
This only makes the bottleneck more of a bottleneck. So, if you want to optimise your experimentation process, then you need to deliberately not optimise before the bottleneck, but you need to invest time in figuring out what the bottleneck is, and then removing it.
Scaling experimentation
When you think about how you scale experimentation in an organisation there’s two parts:
Operational efficiency
Organisational change management
Operational efficiency is about removing bottlenecks and process optimization. How can you test as many ideas as quickly as possible? Change management is getting people to appreciate and understand that experimentation is something that they should be doing.
Both things need to happen at the same time, they go hand in hand.
“If your experimentation process isn’t optimised for efficiency, and things are difficult and take a lot of time, then it’s going to be challenging to convince people to come on the journey with you”.
Rather than focusing on the things that aren’t working, focus on the things that are working, and highlight them.
Show people in the organisation the beacons of shining light – this team is running experiments, they’re having a lot of success doing it, and here’s how they did it.
You absolutely need to split your approach 50/50. On the operational side you’re trying to identify bottlenecks, and on the change management side, you’re trying to identify bright sparks.
2. Identify threats to validity
The ship is always broken!
For every experimentation program, there’s always stuff to fix. The job is never done.
If you’re in the Marketing department, and you’re performing 10 experiments as a side bet, there’s no great threat to the business if you’re making mistakes.
“At Booking.com, when you’re operating experimentation at scale, and nearly all product decisions are supported by experimentation, if you’re not running those experiments with the utmost quality standards, then that is a real threat to the business”.
This places an enormous responsibility on the people supporting the experimentation infrastructure, and the decision-making process. Ultimately, all decisions in the business are going to be supported by experimentation in some way.
If the experimentation team make a mistake in the data collection process, next week 2,000 people are going to be making the wrong decision.
There’s always pressure for the experimentation team to perform.
One of the key mental models that Lukas took away from Booking.com, is that you need to look at all the experiments people are running and identify the threats to validity. You need to address this at scale.
3. Making more decisions a little bit better
There's no such thing as a perfect experiment.
There's only experiments on a continuum between being very well executed, detailed experiments, and winging it off the seat of your pants without an experiment.
In between, there's a large scale of different possibilities.
The intent is not to make a small number of super high-quality decisions, and then wing the rest. This is what generally happens in most organisations.
The number of decisions that the company makes is almost like a constant. It is a fixed thing - the company is going to make decisions, whether you like it or not.
As a decision support team, you must think about how to improve the quality of all decisions, not just a small few, but all of them.
You need to ask, “How do I make all decisions just a tiny bit better, not make a few perfect.”
Making lots of decisions a little bit better has an enormous upside for the business and customers.
If you’re enjoying this article, listen to the full conversation on the Experimentation Masters Podcast
4. Experimentation as product management
Building an experimentation capability in an organisation isn’t all that different from regular product management.
Think about the websites we are building - users come to the website, we try to get them to convert, and they hopefully purchase our products and services.
We spend a lot of time and effort explaining to users why our stuff is the best. Also, we make it super easy for users to flow through our funnel and give us their money. Internally, experimentation isn’t any different, right?
We’re trying to make experimentation as easy as possible, to encourage more people to perform experiments, so that they can make better decisions.
Just like we focus on solving problems for our customers, we should also be thinking about how we can solve problems for our internal stakeholders.
How can you apply the principles of Product Management to your experimentation program?
Who is your target customer?
How big is your market size?
What customer problems are you solving?
What is your customer value proposition?
How do you communicate with your customers?
How do you attract customers?
How do you convert customers?
How do you engage and retain customers?
How do optimise customer experience to make experimentation as easy as possible?
Establishing and scaling an experimentation program in your organisation is no different to a user conversion funnel. You’re trying to get more people to come to your platform.
You can model your experimentation customer acquisition, conversion, and retention in the same way.
5. Establish Communities of Practice to scale experimentation fast
Standardise experimentation language
One of the things that you need to figure out quickly is how you get different teams and tribes to speak the same language.
“Establishing a common and universal language for experimentation is key to scaling experimentation”.
If you really want to scale, then people need to be able to learn from one another. You can only do that if the entire experimentation system is transparent.
Transparency requires standards – the same way of documenting experiments, the same way of performing analysis, the same words to describe metrics etc.
Centralisation and standardisation of knowledge and information is critical for organisational learning to occur at scale.
Establish Communities of Practice
Communities of practice describe how groups of people learn a practical skill from one another.
An example of this is how artisans (tailors, butchers, blacksmith) have transferred skills and knowledge over centuries. Knowledge is transferred from an expert or master to an apprentice through observation and the sequential development of practical skills.
An apprentice tailor doesn’t start with designing a suit, cutting the fabric, and sewing it all together. They may start by sewing buttons on a finished suit jacket. It’s almost like the apprentice works backwards because it’s easier. As the apprentice matures, and gets better, eventually they will design, cut, and sew the fabric.
“There are a lot of parallels with how artisans and experts transfer skills, and how fast-scaling experimentation occurs in organisations”.
Establish Communities of Practice around experimentation so that people can learn from each other through peer-to-peer communities.
Think about this occurring at three distinct levels:
Ring 1 – Experimentation experts
Ring 2 – Experimentation ambassadors
Ring 3 – Peers / Teams / Tribes
Ultimately, you need to get to a position where a new starter joins your organisation, with no practical experience with experimentation, and within weeks, is performing experiments autonomously.
If you can get people to talk to each other, and teach one another, it’s a more scalable approach to improving overall decision quality than having a centralised experimentation authority.
Experimentation ambassadors are the information highway
There should be a very select, small number of people who are the absolute experts on experimentation in an organisation (Ring 1).
The Experimentation Ambassadors (Ring 2) are highly dedicated to experimentation, spending lots of time to understand experimentation in detail and depth.
Experimentation Ambassadors get most of their knowledge from Experimentation Experts (Ring 1), forming the first line of support to the business.
Experimentation Ambassadors are deliberately heterogenous, representing different business units in the organisation.
“Establish ambassadors in business units where there is interest and buy-in for experimentation. This ensures that ambassadors have a clear mandate to spend time working on experimentation. It is their job!”
Experimentation Ambassadors are sponsored and endorsed by a business leader, not the experimentation program manager.
It’s also best to select a broad range of roles and skillsets for ambassadors – UX, Product Managers, Developers, Analysts, Marketers etc.
And then there’s the periphery (Ring 3), individuals, teams, and tribes, who are tangentially involved, and learn through observation and peer-to-peer learning.
Initially, the focus of your experimentation program needs to be on establishing a tight connection between the Experts and Ambassadors.
At the same time, the Ambassadors need to open transparent lines of communication with the business units so that people can start to learn from/about experimentation.
“Ambassadors play an important role in establishing an information highway by creating human connections. They are the local resident, know it all, the person that people can go to if they have any questions”.
When done well, eventually the Ambassador Program can be dissolved, with all experimentation learning and knowledge transfer occurring in the teams, tribes and customer journeys.
Summary
Every organisation wants to perform more experiments.
If you want to scale experimentation in your organisation, think about these things:
Identify and remove friction and bottlenecks in your experimentation flywheel
Growing experimentation through operation efficiency and change management. Both must occur concurrently.
Your primary objective with experimentation is to make lots of business decisions a little bit better
Identify threats to validity in your data collection process
Building an experimentation program is no different to regular product management
To truly scale experimentation, you need to get people to learn peer-to-peer
Remember, there’s no such thing as the perfect experiment. There’s always more work to be done!
Need help with your next experiment?
Whether you’ve never run an experiment before, or you’ve run hundreds, I’m passionate about coaching people to run more effective experiments.
Are you struggling with experimentation in any way?
Let’s talk, and I’ll help you.
References:
Before you finish...
Did you find this article helpful? Before you go, it would be great if you could help us by completing the actions below.
By joining the First Principles community, you’ll never miss an article.
Help make our longform articles stronger. It only takes a minute to complete the NPS survey for this article. Your feedback helps me make each article better.
Or share this article by simply copy and pasting the URL from your browser.