Increasing experimentation program sophistication & maturity
First Principles were engaged to help increase the maturity and sophistication of this experimentation program in preparation for a transition to machine learning model driven experimentation.
Before partnering with First Principles, this organisation had some challenges with their experimentation program:
Processes were manual and highly complex
Experimentation throughput was low
Experimentation data was untrustworthy
Many experiments didn’t move the needle
Date
Oct 2023 - Feb 2024
Client
Services delivered
- Experimentation strategic advisory
- Detailed experimentation solution recommendations
- Experimentation program transition plan & roadmap
The business opportunity
This company is a market-leading online sports wagering company in Australia, with a market share of 50% and revenues of $2.25B. The business has developed some foundational capabilities with experimentation over a period of four years, conducting approximately 200+ experiments per annum.
It currently takes the experimentation team a long time to perform experiments, with all experiments executed manually. The business was seeking greater agility and flexibility to learn from customers faster. Conducting individualised and personalised experiments to highly targeted customer segments was challenging.
The team were seeking to increase experimentation program maturity and sophistication to grow market leadership position. Key to this goal is being able to allocate generosity investment more effectively through highly targeted, personalised offers, ensuring better returns on customer investment.
The key objective of the project was to define a new, future state experimentation operating model to support successful machine learning model driven experimentation.
The solution
This project was strategic advisory consulting, with First Principles required to develop a series of solutions and recommendations to increase experimentation quality, increase experimentation speed and improve experimentation program governance.
The project was run over a five-month period.
There were four phases of the project:
Strategy & requirements – establish project foundations & define experimentation program strategy
Insights & assessment – conduct discovery and analysis to understand current state landscape
Design & justification – design and develop the future state operating model
Recommendations & roadmap – experimentation program transformation pathway defined
Strategy & requirements
Foundations set in place to ensure a disciplined and successful project – Project Scope & Deliverables, Project Timelines, Stakeholder Engagement Map & Kick-Off Meeting
Experimentation program Strategy & Mission defined through a collaborative workshop with key stakeholders – Mission, Strategic Alignment, Purpose, Objectives, Values/Behaviours, Program Performance Metrics
Insights & assessment
16 teams were engaged and 33 interviews conducted with key stakeholders to analyse and understand the current state environment – industry, business operations & experimentation processes, systems & tools
Organisational capability assessment was conducted to identify Organisational Strengths & Weaknesses
13 high-priority, high-impact improvement opportunities identified for the experimentation program
Design & justification
11 key stakeholders were engaged for business requirements elicitation. 84 raw business requirements were captured
List of 34 synthesised business requirements were reviewed and prioritised collaboratively with stakeholders
21 requirements prioritised as High Priority for the experimentation program
Collaborative workshop conducted to determine fit-for-purpose Experimentation Team Structure
Detailed solution recommendations were prepared to address all high-priority improvement opportunities and business requirements
Solution design recommendations were socialised with key stakeholders for feedback
GAP Analysis conducted to understand transition pathway from Current State to Future State
Risk Analysis conducted - 9 high-impact risks identified
Recommendations & roadmap
Solution design recommendations further socialised with key stakeholders for feasibility & refinement
Workshops conducted with key stakeholders to communicate solution design recommendations
Blueprint & rollout plan formulated for transition pathway to future state operating model
The results
22 detailed solution design recommendations prepared to solve for high-priority, high-impact improvement opportunities
3-year transition plan defined for solution implementation, including identification of key dependencies and sequencing of initiatives
New Future State experimentation program operating model defined - team structure, processes, systems, governance, roles & responsibilities
Key learnings
Model testing
Machine learning models can and should be tested in production like anything else. Testing machine learning models should be viewed like a product change, with a rigorous A/B testing approach applied.
High-value customers
Not testing regularly on high-value customer segments introduces business risk. Changes should be tested on all customers who are receiving the change. Not experimenting on high-value customers limits ability to understand customer drivers for more impactful innovation.
Interaction effects
Test interactions are rare and generally not worth worrying about provided monitoring is in place. Trying to isolate tests is far less valuable than the testing velocity achieved by allowing overlapping tests to run in parallel.
Experimentation autonomy
Experimentation programs must have the independence and autonomy to pursue broader organisational learning objectives. Separating experimentation from BAU planning processes accelerates customer value creation and delivery.