Continuous products need a continuous measurement mindset
In the past century, companies operated in a context where changing existing products was extremely difficult. Once a physical product was designed, all the specifications sent to the operations team, and the manufacturing process defined, making changes to the product was often incredibly expensive and painful. In that manufacturing world, most decisions were irreversible, and iterating existing products in the field was almost impossible.
In the digital world, this is no longer true. With modern Cloud tech stacks and DevOps tools, the code can be released and updated multiple times a day, making digital products ‘continuous’ and ever-evolving. This means that the investment cost associated with product decisions is significantly lower, as many more elements can be released, tested and modified quickly and easily.
The continuous form of today’s digital products has made it possible to adopt fast, responsive and data-driven product approaches, where teams work in quick 'ship and learn' iterations that rely on users’ behavioural data to measure the results.
In this context, the core definition of what ‘success’ means for a product team is changing. In a world where product plans and initial specifications were the only sources of truth, success meant that the feature was released according to specifications, on time and on budget. In today’s digital world, product managers are instead judged on their ability to deliver continuous value to users. Therefore, success needs to be defined in a much more user-centred, responsive way. ‘Done’ as ‘we developed and shipped the feature’ does not mean ‘successful’ anymore.
The release of a new feature is just the beginning of the journey.
Decentralised decision-making and product success metrics
Due to the nature of industrial products, past management practices prioritised scrupulous planning (requirement analysis, detailed long-term product specifications, business cases) over measuring the performance of existing products. This also generated a risk-averse mindset to avoid costly mistakes; and a centralised slow decision-making process, involving many sign-offs and gatekeepers.
Today, teams that build products in a responsive and continuous manner should not be trapped in pre-defined, long-term roadmaps and complex sign-off processes. Long-term plans cannot keep up with fast, empirical learning. A flexible measurement system is required – one that allows leaders to set broad objectives and leaves autonomy for fast, evidence-based decisions on specific product aspects to the team.
The idea of independent product teams and decentralised decision-making emerged in the last few years and has been popularised by some of the best product-led companies in the world. For instance, this is Spotify’s take on the matter.
When teams operate autonomously, product metrics become an essential management and accountability mechanism for product leaders:
- Autonomy is created by ensuring that leaders define the overall vision and intent of the product with strategic metrics, and then cascade that down to each team with tactical sub-metrics, giving them the autonomy to figure out how they will achieve the vision with fast continuous ‘ship and learn’ iterations
- Product teams need to understand that they are not just a cost centre that develops and ships code as fast and efficiently as possible, but they are responsible for taking ownership and achieving the final business outcome.
Defining a shared framework of success metrics is the most important management tool for product leaders today.
This also means that companies need to set up systems to measure product analytics and provide easy, open access to data across the company, to help the teams understand what is and isn’t working – and iterate quickly.
A responsive, user-centred definition of ‘success’
Setting up a product metrics framework that defines a shared vision and measures the progress is hard. Modern digital products are constantly evolving and therefore cannot be judged by the same evaluation criteria that businesses have been using for decades.
At the core of our product metrics framework is the distinction between three different types of metrics: output, outcome and impact.
- Output metrics, i.e. ‘we have finished building the feature/product’. These metrics measure process performance (time, efficiency) and technical quality of the product.
- Outcome metrics, i.e. ‘the feature/product has generated the customer behaviour change we wanted’. These metrics measure how successful the product is in helping customers achieve their desired outcomes and in changing their behaviours.
- Impact metrics, i.e. ‘the feature/product has generated the financial return we expected to achieve’. These metrics measure how successful the product is in generating a financial return for the company.
While companies should measure all three types of metrics, they are not all equal.
If you work in an Agile and Scrum team, you will certainly hear people talk about output metrics such as story points, velocity, defects and bugs. These are clear, quantifiable, widely used metrics to assess an agile team’s performance, efficiency and speed. However, judging a product team simply by measuring the velocity and the number of features developed is like evaluating a patient’s health by measuring the amount of food they had, and how quickly they ate it. It makes no sense. As our Technology Director Mark Rodseth previously explained on our blog, when success is defined mainly by output metrics, product teams get easily side-tracked.
Impact metrics can be misleading too. It's important for product teams to consider impact metrics at regular intervals, as they help connect product decisions with business outcomes. It’s common for teams to focus on metrics such as cost per acquisition, average sales price, cost savings, monthly recurring revenues or Lifetime Value (LTV). Nevertheless, impact metrics are lagging indicators of success – they can help measure past performance, but are unfit to predict the future. In the long-term, focusing solely on business value and forgetting customer value may drive teams in the wrong direction.
This is why outcome metrics are crucial. Companies that embrace the modern lean, responsive, 'ship and learn' product mindset need to have sources of truth that help teams constantly align decisions with customer needs and behaviours – and help them make more informed decisions about what to build. Outcome metrics are user-centred indicators that measure the value delivered to customers, based on changes in customer behaviour, attitudes, or knowledge, or in other words, stages in the process you expect users to reach along the journey.
The best outcome metrics are also leading indicators of future financial impact. Average time per session, for instance, is a great metric for an app with a revenue model based on advertising. On one hand, it quantitatively measures a positive change in customer behaviour (e.g. users spend more time in the app). On the other hand, it’s a leading indicator of the revenue (i.e. more time in-app = more exposure to ads = higher revenue). If you're interested to know more about outcome metrics, check out this recent podcast with Josh Seiden – the author of Outcomes Over Output.
The process of re-focusing your teams on outcome metrics is not straightforward. Organisations are usually very good at defining the high-level business goals and impact metrics they want to achieve, such as reducing costs by 10% or increasing customer loyalty by 5%. But after that, they often jump directly to the systems and features they need to build, skipping the ‘user outcome’ part: ‘We will reduce costs by implementing a new mobile app for the staff’.
The direct connection between the impact metrics and ‘the number of features to implement’ is what generates the idea that ‘done means successful’. By contrast, decomposing the business goals into outcome metrics helps the teams to focus on what really matters and what leads to financial success, i.e. affecting user behaviour to drive positive change. It also helps to translate high-level, intangible business objectives into specific achievable goals and distribute them across different teams.
In the next article, we will discuss how to organise these metrics in a comprehensive strategic framework that tracks progress against your product goals. We will also discuss different types of measurement systems and provide examples of great success metrics at each step of the product journey. Stay tuned!
Wondering which tools you need to implement? Read our recent article on how you can build an effective product analytics stack that streamlines access to the most relevant metrics.
Author: Nicola Bertazzoni