Six Lessons From The Award Winning bol.com Experimentation Program
“Even with the small number of tests we did in 2018, we were able to generate an additional €100M in revenue, which is a lot of money. It was about low hanging fruit, because we only put first iterations live and never tried to iterate. This opened the eyes of the board and senior managers”
You might also enjoy these articles:
United Nations - Solving The World’s Toughest Challenges With Experimentation
Hacking IKEA : How To Use The Infiltrator Technique For Product Experimentation
Introduction
When it comes to eCommerce platforms, experimentation is becoming more and more important to increase conversion rates, grow revenues and improve customer experience.
One example of a company that’s developed and grown in its approach to experimentation is the Dutch eCommerce titan bol.com. bol.com is the largest eCommerce business in the Netherlands, generating €5.5B in revenue in 2021, serving 12+ million customers more than 34 million products from 34,000 retailers.
Denise Visser is the captain of Team Experimentation. For the past three years, she has worked as Experimentation Product Manager at bol.com, growing experimentation in the business from less than 100 experiments in 2018 to a team that became the winner of the Experimentation Culture awards in 2021.
We were lucky enough to catch up with Denise to pick her brain about working at the coalface of experimentation every day.
Throughout this conversation, we looked at how to define success in business, the value of organisation learning and how to establish and grow an experimentation program.
In this article we discuss:
Success isn’t binary
Experimentation provides focus
Testing without knowing
Learning before earning
Working smarter, not harder
Start small, start local
1. Success isn’t binary
We need to start by asking the question – What does success look like?
There are many ways to measure the success of an experimentation program - experiments per month, new revenue, budget saved etc.
“We’re all aiming for better achievements. But how do you know when good is good enough? What are the numbers of your performance today, last week, last month, last year? Can you explain the differences between those numbers? What's your plan ahead?”.
However, success shouldn’t be measured in a binary fashion. Denise argues that it’s more important to understand how you measure success and how the success was achieved.
The ultimate marker of success is when your product meets the needs of your users.
“Experimentation is not just a hit and run mission on revenue. You always want to be learning from your customers so that you can understand how to serve them better. Learning from experiments is always far more valuable, even if the results don’t immediately translate into revenue.”
Experimentation plays an important role in helping you to understand the effects and impacts of your efforts – how your inputs affect outputs for customers.
2. Experimentation makes you smarter
Your list of tasks and ideas is always longer than available time. Time is scarce, so tasks need to fall off the list.
“Experimentation helps you to focus on the tasks that are going to have the biggest impact on your users, so you can maximise customer behaviour change”.
Experimentation helps your team to become smarter. Teams can learn from their experiments where they should be spending their time to have the greatest impact.
Tasks are often prioritised based on who shouts the loudest. Experimentation changes this by enabling you to bring proof to the table, ensuring task prioritisation is customer focussed.
The smarter your team gets, the more effective the team gets.
3. Testing without knowing
Too often, the launch of new features and products comes with celebration and fanfare.
Denise suggests that, rather than celebrating a new feature that went live, “we should put more effort into learning about the actual impacts of the new functionality or adjusted functionality on the end user”.
That’s when you start knowing and you start learning.
A key piece of advice from Denise:
Keep track of all the releases in your company and keep track of what the impact was on user behaviour, and success metrics of the company. Then you can learn from the past which hypotheses turned out to be true, and which not.
4. Learning before earning
Microsoft has previously reported that more than 60% of their experiments had either neutral or negative impact on users.
Think about that for a moment - only 30% of experiments from one of the world’s leading companies yield a positive impact for users.
Denise thinks that every experiment is a success because you can learn something from it. Emphasize learning before earning.
Three important tips:
1. There’s no such thing as a failed experiment, only experiments with unexpected outcomes
2. Every time you perform an experiment you learn more - you cannot learn less
3. Don’t stop testing your hypothesis after one experiment - dig deeper to come up with more possibilities
If you’re enjoying this article, listen to the podcast conversation on the Experimentation Masters Podcast
5. Smarter, not harder
There are many different types of parallel customer research being performed in businesses every day – experiments, quantitative research, and qualitative research. It’s easy to lose track of all the valuable customer insights.
Gaining insight makes you smarter, also making it easy to work smarter. You can learn faster and make better quality decisions.
Denise recommends establishing a Customer Research Library or Learning Centre to centralise storage and management of all customer research and insights.
Where customer research becomes most powerful is when it can be analysed at aggregate level, providing a 360-degree perspective on the customer problem you’re trying to solve.
If customer research is stored in network directories or email inboxes it renders the insights useless.
6. Start small, start local
(A). EARLY DAYS:
There was no culture of experimentation at bol.com.
While proudly an innovative company, bol.com was guilty of pushing features live, not evaluating the impacts of feature releases, then focussing on the next new feature release.
While impatience and enthusiasm often overrode business decisions, the organisation still had great success, enjoying year on year growth.
(B). ESTABLISHING THE PROGRAM:
The bol.com experimentation program started small and localised. There were no dedicated resources for experimentation in the business at the time.
“We started with a group of enthusiasts, some business analysts from the webshop and the app. We had some web analysts in our team and a developer also joined the club. We made use of the scarce experimentation resources that we had”.
The experimentation team set a goal of performing 100 experiments in 2018. The focus of these early experiments was on organisational learning, with less emphasis on growth.
With only small, front-end adjustments possible, the bol.com experimentation team had a deliberate strategy to focus on quick wins (I.e., copy changes, re-ordering page components etc.)
The team quickly realised that performing 100 experiments was very difficult.
“Even with the number of tests we did in 2018, we were able to generate an additional €100M in revenue, which is a lot of money. It was about low hanging fruit, because we only put first iterations live and never tried to iterate. This opened the eyes of the board and senior managers”.
(C). SCALING THE PROGRAM
Fast forward to 2022 and the bol.com experimentation team is now performing more than 80 experiments per month.
While the experimentation team started out with a small scope, the business is now seeking to expand experimentation across the entire bol.com platform.
With an increasing scope for experimentation, comes more complexity. There are now greater dependencies between teams and cross-functional collaboration required to continue to scale experimentation.
Summary
Just make a start. Don’t get too hung up on trying to prove whether experimentation is valid for your company or not.
Don’t give up after the first round of experiments. The first experiments are always the hardest.
When you’re starting out with experimentation, there is always low-hanging fruit. Focus on early quick wins to deliver impact and create momentum with experimentation.
Most importantly, you need to tell everybody what you’re doing. Communicating successes works wonders to develop enthusiasm and interest in your experimentation efforts.
“Communicating to your audience is important. If they are aware of what you want to achieve, and that is not only winners, but you want to achieve learnings, then it’s easier to continue if the first experiments fail”.
Need help with your next experiment?
Whether you’ve never run an experiment before, or you’ve run hundreds, I’m passionate about coaching people to run more effective experiments.
Are you struggling with experimentation in any way?
Let’s talk, and I’ll help you.
References:
Before you finish...
Did you find this article helpful? Before you go, it would be great if you could help us by completing the actions below.
By joining the First Principles community, you’ll never miss an article.
Help make our longform articles stronger. It only takes a minute to complete the NPS survey for this article. Your feedback helps me make each article better.
Or share this article by simply copy and pasting the URL from your browser.