How To Avoid Being A Data-Driven Product Manager
Following the data in blind faith is dangerous. Data does not provide risk-free, imagination-free decision making. Being 100% data-driven can produce lazy, blinkered decisions.
You may also enjoy these articles:
Introduction
In an age of big data, it feels like everything must be data driven. Data has become omnipresent in every element of our work, personal and family life.
Data-driven has become much more than a buzzword. It has become a deeply entrenched part of business lexicon.
For Product Managers, data has become a driving force for decision-making, improving customer experiences and generating new ways to solve problems.
Being data-driven is not all upside though. Having access to more data provides a false sense of confidence and security, a feeling of comfort, like we have all the answers.
This can leave the impression that there’s nothing new to learn. We’ve got all we need to charge on launching that new product or building that new feature.
This can result in lazy decision-making, where the data is followed in blind faith, no questions asked, deferring responsibility to the data if things go wrong.
A better approach is to be data informed. This approach is a nod to decision-making requiring multiple data sources to be evaluated and interpreted.
Good product managers use a combination of subjective and objective data inputs. Imagination and creativity are still critical to solving complex business problems.
In this article we’ll discuss the following:
Why it’s difficult to turn our beliefs and instincts off
Why there’s always uncertainty in our decisions
How it’s easy to become a slave to data
How analysing data is akin to rowing in a lake
10 practical tips to ensure you’re being data-informed
1. Your beliefs aren’t a tap, you can’t just turn them off
There’s this growing perception that Product Managers see their decision-making process more reliant on data than instinct or imagination.
More than ever before. Fact.
There’s much talk about eradicating our beliefs and intuition from the decision-making process.
Only, it’s not possible.
It’s not how the brain is hardwired to work.
We can’t just flick a switch and magically turn off our beliefs, emotions, and feelings.
Whether we like to acknowledge it or not, our beliefs and intuition are always sitting there, playing away in the background.
Sensing and feeling will always be central to all decisions that we make.
“The overall philosophy I think about is harmonising a balance of signals gathered from data to better inform your intuition. I think there’s always an element of interpretation and insight gathering that needs to happen, even with data” – Manosai Eerabathini, Google
It’s just that we don’t want to lead with emotions in the decision-making process.
Being 100% subjective in your decision-making process will undoubtedly lead to sub-optimal outcomes. You can’t guess your way to success.
Decision-making will always be a combination of objectivity and subjectivity.
“Data helps provide objectiveness, so I think being data-driven is about looking at the data first, while still acknowledging that there’s always a place for gut instinct and experiences to drive decision-making” – Beatrice Fabris, Mimecast
Weaving subjectivity into your decision-making process won’t make your decisions impure.
Don’t worry, you’re not going to be banished to a deserted island of product management misfits.
If you’re thinking process is sound, it is still highly defensible.
2. It’s not possible to have all of the answers
Business is more like poker than chess
Chess is commonly used as an analogy for business. This doesn’t make sense.
In a game of chess, based on the position of a piece on the board, and the location of all the remaining pieces, a good player should know all possible moves and strategies.
An inexperienced player would never beat a Grand Master. Experience and strategic know-how always win hands down.
Poker, on the other hand, is a much better analogy for business. I’ve previously written about this here, and here.
Poker involves a combination of bluff, luck, strategy, and decision-making, all in the presence of unknown or incomplete information.
It’s possible for an amateur player to beat a World Champion given the right cards.
Decision-making occurs under conditions of uncertainty
More than ever, business decision-making is required to occur in the absence of complete information.
The likelihood of Known Unknowns and Unknown Unknowns is increasing.
Businesses can’t tread water until all the information is available. The opportunity-cost is too high.
We need to become more comfortable with “I don’t know”.
“Decision-making occurs under conditions of uncertainty”.
That’s why using probabilistic thinking, to think in terms of probabilities, is a better mechanism to understand confidence levels.
“In real-world decision-making, even if you have amazing data processing capacity, you never know everything you need to know to answer the question”.
No business ever has all the data it requires. You’ll always want more data.
The desire for data is infinite.
There’s this weird psychological safety need to keep diving deeper on analysis, cutting the data in different ways, requiring more input data sources etc.
This stems from “wanting to be right” and not making mistakes.
This behaviour serves as an audit trail of sorts, covering ass if things go wrong.
You can always point back to the breadth of data analysis undertaken to absolve of any responsibility.
Right amount of data, right time
At some point you need to stop and ask, “What information do I need?” and “What information do I want?”
What information is required to make a decision that I can act on tomorrow, the next day or this week?
“That mythical, unicorn like kernel of information that’s going to provide decision-making nirvana, doesn’t exist”.
The faster that you can make decisions, the faster that you can create and deliver value to your customers. Decision-making is your competitive advantage.
Product Managers drive the bus
In every case, there’s a Product Manager sitting behind the wheel, driving the data bus.
The Product Manager is ultimately in charge of determining what data and how much data. There’s still no magical algorithm to tell us what decision to make. Maybe one day.
You’re still required to decide. That’s unavoidable. That’s your job.
For the foreseeable future you’re going to be required to analyse and interpret data, make meaning and inferences, then decide on a course of action.
Decision-making for your product is not like a manufacturing production line. It’s not as simple as inputs in (data) and an output (decision) is spit out the other end.
Sure, some product related questions are straightforward and binary. How many signups did we have last month? Easy!
As soon as you need to start understanding the “why” of customer behaviour, you inevitably need to leverage qualitative and quantitative research in combination to develop a holistic picture.
This is where things start to get more complex, and you can end up down a worm hole for weeks or months if you’re not disciplined and structured.
3. Being held prisoner by data
The pressure on product teams to get things right is next level.
All companies want more growth. Product teams are pivotal to growth. Yep, I get it.
However, in the pursuit of relentless growth and business advancement, it’s like we’ve almost become handcuffed by data.
We’re captive to the data, being held prisoner.
Data should not be used as an out clause.
An over-reliance on data is just a way to deflect any potential blame. This results in risk averse, imagination free decision-making.
Imagination and creativity are still critical to solving complex business problems.
Business problem-solving is not a straight line, linear process of rationale deduction.
“Sequential logic and reasoning can only take you so far. Complex decision-making in business, science and physics is an abductive process, with inference and interpretation involved ”.
As previously stated, information is frequently missing or unavailable.
If your business is 100% insistent on only making scientific, data-driven decisions, you could be missing out on the brilliant, transformative ideas.
Data-informed is a better mantra than data-driven.
Data-driven at all costs is disingenuous.
This approach can create organisational blind spots and future problems.
There is a time and place for scientific decision-making in business.
4. Row in the lake, but don’t exhaust yourself
Data is like a big lake
Let’s think about a lake as a metaphor for data.
You’re out rowing in your boat on a lovely Sunday afternoon, exploring this beautiful, big mass of water (data).
There’s many different streams or rivers that feed in to fill up the lake. The lake becomes this broader ecosystem.
Think of the big lake like an aggregation of all the data and information sources in your business.
The streams and rivers that feed into the lake represent all the different qualitative and quantitative data sources.
Explore, but don’t get lost
Understanding your customer or business problem is a process of search and discovery, just like exploring in your rowboat.
You need to row your boat up some of the rivers and streams to explore, to see what’s there, understanding if there’s anything of value or interest.
However, you’re not going to row your boat up and down, in and out, of every one of those individual rivers or streams.
“You need to understand what’s relevant and not relevant”.
A skilled Product Manager can decipher what data is required and what data is irrelevant. Then, make sense of it.
“In a world deluged by irrelevant information, clarity is power” - Yuval Noah Harari
That would take forever. It’s not a good use of your time.
You could get lost. And tired.
While there’s a mass of water that you could potentially cover in your rowboat, you’re focussed with your efforts and time.
Rowing a boat all day is tiring. You don’t just keep rowing your boat infinitely because you can.
You only row far enough and long enough to meet the objectives you defined before you set out on your journey.
More is not better.
Once you’ve satisfied your objectives, you call it a day, moving on to the next challenge – finding a good place to grab a drink and meal.
Happy days!
Enough rowing … you get the drift.
Proceed based on an implied level of confidence
When you have sufficient data to decide, make the call with an implicit level of confidence, knowing that you can always gather more information later to improve your confidence level.
“It’s always better to make lots of reasonably informed decisions, reducing opportunity-cost and the cost of being wrong, than delaying decision-making further “ - John Cutler
Advancing the confidence level in your idea or hypothesis evolves over time, it’s not static.
We have a hypothesis, perform an action, we learn something, we keep doing more of what works and we do less of what’s not working. Repeat.
5. 10 practical tips that you can apply in your role ..
Set objectives - clarify your learning objectives upfront. What do you want to know? What do you need to know?
Outputs - be clear on what type of data is required to answer your question. Simple – quantitative. Complex user behaviour – multiple data inputs combining a mix of quantitative and qualitative data
Speed - shorten your horizon – what do I absolutely need to know so that I can move forward tomorrow, or the next day?
Probabilities - be prepared to move forward with an implicit level of confidence, knowing that you can increase confidence levels over time
Uncertainty - embrace and accept that information will always be incomplete or missing. No business ever has all the data
Sizing - always endeavour to use the right amount of data. Know when enough is enough. More is not better
Signals - there is no one data point that is going to provide decision-making nirvana. Look for signals and leading indicators to provide directionality, rather than aiming for a "unicorn” data-point
Subjectivity - there’s always a place for subjectivity in decision-making. Complex business problems can’t be solved using a linear process of deductive reasoning
Responsibility - don’t fall into the trap of blaming the data if things go wrong. Understand why. Accept responsibility
Spin doctoring - don’t spin the data to create a narrative that’s not there. That’s not cool. It’s your duty as Product Manager to always paint a complete and accurate representation of the data
Conclusion
The desire for data in business has become insatiable. Data is everywhere. Guessing is not a sustainable strategy for organisational success.
However, there’s no such thing as risk-free, imagination free decision-making.
Following the data in blind faith is dangerous. We’ve become captive to data.
When things go wrong, the data becomes a martyr. We followed the data. What more can you ask for?
Being data-driven can produce lazy, blinkered decision-making.
The advancement of your product is a non-linear process. It’s not a straight-line process of sequential reasoning and deduction.
Good product managers use a combination of subjective and objective data inputs.
They understand that product management requires inputs from qualitative and quantitative sources to validate and refine hypotheses.
There’s always inference, and interpretation involved during the product lifecycle. We’re constantly required to make decisions with no information, or missing information.
Imagination and creativity are still critical to solving complex business problems.
Need help with your next experiment?
Whether you’ve never run an experiment before, or you’ve run hundreds, I’m passionate about coaching people to run more effective experiments.
Are you struggling with experimentation in any way?
Let’s talk, and I’ll help you.
References:
Before you finish...
Did you find this article helpful? Before you go, it would be great if you could help us by completing the actions below.
By joining the First Principles community, you’ll never miss an article.
Help make our longform articles stronger. It only takes a minute to complete the NPS survey for this article. Your feedback helps me make each article better.
Or share this article by simply copy and pasting the URL from your browser.