Product Metrics: Identifying Opportunity and Inefficiency

October 13, 2016 / Calvin Marshall

On the business side of product management, an area that typically gets short shrift is the creation, measurement and review of metrics related to a product effort.

In many companies, Product Management organizations focus on leading and launching product-specific initiatives. Product Managers combine a mix of technology, industry and business knowledge to move initiatives forward.

Note: throughout this entry, a “product effort” will be used interchangeably to highlight a ‘new product or service’, a ‘new feature’ or an ‘enhancement of an existing feature’.

Product metrics are more than simply interesting insights. Metrics are not captured merely to present a ‘Product Dashboard’ and show the current status of products in-progress. Whether justifying a stance or supporting strategic and tactical changes, product metrics should be viewed as essential to decision making. Product metrics should be actionable. A Product Manager and the Product Leadership team should regularly review product-specific metrics. The type of metrics measured and action steps to take depends on the industry and the issue.

Here are some obvious, and perhaps not-so-obvious, metrics product teams should consider tracking for better management of the product portfolio.

Value achieved

 At first glance, the value a product effort achieves may seem obvious, however many product teams don’t review the basics. These teams see it as someone else’s responsibility. Whether a product team or another internal team measures the bottom-line value of a new product introduction, this is one area that rarely gets measured at the product level. Yet Product Leadership and Executive teams want to know how a new product or a foray into a new market is doing.  

While metrics for a new feature are difficult to measure, there can be tremendous value in confirming whether a new feature offers greater volume, generates more revenue or reduces attrition. Likewise with a new enhancement, teams could measure customer satisfaction or increased interaction. Measuring bottom-line metrics like volume, revenue and/or cost savings should be expected. 

Leadership teams should be asking: ‘After all, we spent money developing the capability. What value did we achieve?’

Total Cost at a granular level

Within IT organizations, many companies are diligent in tracking the costs associated with software development projects to measure the cost to deliver a project. Given that launching a product effort is a cross-functional, collaborative activity, all costs including those outside of IT should be measured.  A product effort may offer increased volume, increased revenue or a cost reduction —what is the total cost vs. the product effort’s benefit.

A product team, or groups outside of the product team, may create a business case to gather approval and resources for major product initiatives. For product efforts that are smaller in scope, the business case may be reduced to a ‘back of the envelope’ estimate or a ‘scientific wild-a__ guess’ (aka: ‘swag’).

To truly determine if a product effort is worth pursuing, product teams must review efforts with rigor and at their most granular level. Often times, leadership teams are primarily focused on metrics at an aggregate level (e.g. line of business; domestic vs. international; air vs. ground; public vs. private). Finance or Playbooking teams are tasked with identifying value at the top line level. Product managers are tasked with leading product efforts; they should be focused on product-level metrics. Since each effort can be different, especially for manufactured or bundled services offerings, products should be measured at their most granular—at the SKU level or individual service level.

In one organization, I became the product manager for a product that was to be launched on a new platform. Existing customers were to be migrated from their current platform to the new platform. The previous product manager had allowed the new platform to be released prematurely and without key customer features. Having metrics available allowed me to prioritize features for development and set the roadmap for the new product. I was able to answer the cost/benefit riddle with a high degree of comfort. This allowed me to focus time and resources on the features that impacted the largest number of customers.

Duration from Initiation to Launch

No product effort is the same—each takes a varying amount of time from the moment an effort is begun until the product launch. Capturing this information is valuable from a historical view of a product effort. Consider tracking kickoff date, relevant milestones <those which pertain to your particular industry>, expected launch date, actual launch date and delay reason.

Product Leadership and product teams can get a sense of the time it takes to launch a product effort and the length of time between significant milestones. If a previous, similar effort has been measured for some duration between initiation and launch, this can offer greater forecasting accuracy. (In some organizations, a project manager may be measuring this. Often, the project manager is responsible solely for I.T. related activities).

The real value that I have found has been in identifying issues that cause late product launch dates. For internal delays, I’ve found unrealistic time estimation has been the culprit. Other times, there were resource availability issues—unforeseen situations that came up after a product effort was already scoped. Examples of external delays include: the complexity of manufacturing or the over-promise of a vendor. The value lies in identifying root causes so that potential issues can be identified and their impact can be reduced or avoided.

Forecast vs Actual

Similar to total cost, the ability to forecast expected volume and revenue at a granular level and measure against actual product efforts is beneficial. The ability to take a backward view of key metrics gives an organization a reality check. Part of the business case or any estimate of future results is most valuable when captured in detail. Rather than an overall year estimate, consider a monthly focus that includes a period where a product is allowed to ramp-up volume or usage. Forecasted versus actual can also be used throughout the product effort and validated with actual results. The delta between the forecast vs actual helps identify flaws in processes or a shortfall in expectations.

At a previous company, product sales were highly seasonal. The product team looked at the conversion results on a daily basis to determine how actual product placement fared against the product forecast. With the product launched, the team pushed customer facing teams to generate and convert a higher number of leads into actual sales. In future seasons, the organization held the product team playbookable for product expectations. Actual performance became more closely aligned to the product forecast and focused on fact rather than fiction.

When products don’t achieve their goals, consider the genesis of the product effort. How was it conceived? What factors determined product effort prioritization? Companies often make product decisions by executive decree or because of their most vocal customers. Along with market data, strong product metrics can help product teams support new opportunities by using historical information as a guide to provide better estimates and identify blockages that prevent success.

Based on your industry, various metrics may be important to track. The ‘right’ product metrics help identify areas of inefficiency and opportunity. This allows product teams to improve products for customers and bottom-line results for organizations. Addressing questions like the ones below may offer greater product portfolio management.

• Is the product team meeting the expectations of launched product efforts?
• What are the total costs involved? What is the cost vs. the benefit?
• How long does it take to launch a product? Are launch dates met? If no, why not?
• How close are forecast metrics vs. actual?

Fundamentally, metrics identify both failure and success in your product efforts. Examine the deltas between what was planned and what was achieved to identify areas for improvement in your processes from idea to launch and beyond.