Over the last year, I’ve had a number of conversations about measuring the impact of product — which is tough. But at a recent presentation to the INMA board of directors, I was asked an even harder question: How do you benchmark a product? 

The easy answer is whether people or advertising are paying for it and whether we are the leaders in the field. Of course it’s not that simple. People aren’t just paying for the experience; they are paying for the content. And decoupling the two is nigh on impossible. 

Product is the sum of all the parts. Different features within a product are built and optimised to do different things, to solve a unique user problem. So most experiences are designed to appeal to a subset of users. 

The Netflix definition of product is something that “delights customers in margin-enhancing ways.” So perhaps we should look at the “customer delight.” That could be done via app store rating for a mobile product or something like an NPS score. I know at least one media organisation is working on their own customer satisfaction score. 

Or we could rely on the second part of that definition: If users like the experience, they will pay for it. The benchmark is therefore similar experiences, our competitors. Do people love our product enough to pay for it over other similar products? 

Or perhaps we go deeper, measuring a number of different features and journeys by doing A/B tests. It would take a relatively complex formula to bring these together into some kind of score that decouples the experience from the content. But this is a lot of work and would come at the opportunity cost of working on the actual product.

I asked Karl Oskar Teien about how he thinks about this for paid products at Schibsted.  He says he’s interested in “how people are approaching the intangible … the quality of a product and how that affects loyalty over time. Because it isn’t measurable with just basic click-through rates or other kind of leading indicators of engagement that we can use.” 

It’s an interesting question and one that we as an industry don’t have an answer for. Which is also why product leaders and teams often focus on demonstrating impact on specific projects and results when they are asked about this internally.  

How do you benchmark product at your organisation? I’d love to hear about it. E-mail me at jodie.hopperton@INMA.org.

If you’d like to subscribe to my bi-weekly newsletter, INMA members can do so here.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .