Nearly all industry leaders think product is important (93%) — but fewer than half feel that the role of product is actually understood, according to a Reuters Institute annual trends and predictions report.

This chasm is the reason the INMA Product Initiative exists.

Product is constantly evolving, says Jodie Hopperton, lead of the initiative. In a Meet-Up for INMA members on Wednesday, Hopperton defined product in the news media world as a cross-organisation function, working closely with content and technology, which balances the consumer experience with business objectives.

The focus of the Meet-Up was to help define product success and move forward a shared understanding of goals and desired outcomes, along with key metrics everyone in an organisation can buy into. INMA members can subscribe to Hopperton’s bi-weekly newsletter here.  

What is product success?

Traditionally there have been two types of customers for news media organisations: readers and advertising customers. In today’s modern media landscape of multiple ways of delivering and consuming content, however, readers have evolved into a new type of customer: the audience.

Marcel Semmler, global head of technology publishing at Bauer in Germany, told INMA members today’s audience may be reading content — but they may also be listening to it, viewing it on video, interacting with it in apps and other formats, as well as experiencing it fully in a way that goes beyond just reading.

In addition, the advertising customers have also evolved into business customers.

“We’re looking still at advertising, but we’re also looking at B2B customers, we’re looking at affiliates,” Semmler said. “In addition, we have a third type of customer we often neglect but we need to look at in terms of a good product development, which is our internal users. In order to really drive product success, we have to look at all three dimensions.”

All three customers — audience, business customers, and internal users — are relevant. And product success means creating value for all three.

Value drivers vs. value indicators

Semmler emphasised the difference between value drivers and value indicators. Value drivers include:

• Culture
• Processes
• Organisation
• Transparency
• Data
• Behaviours
• Actions

In fact, everything a company does is a value driver.  Value indicators, on the other hand, include:

• Content value
• Net promoter score
• Involvement of users
• Adoption rate of new products and technologies
• Traffic
• User engagement metrics
• Turnover
• EBIT

Basically, everything that could be considered a KPI that helps measure generated value is an indicator. Many organisations focus mostly on the value indicators, but Semmler said it’s more important to manage the value drivers than the indicators: “We need to look at the root of the problem [value drivers], and not the weed that is coming out on top of it [value indicators].”

Product success is more than technology

When it comes to managing the user journey for the three sets of customers, Semmler said this should focus not only on the technology side.

“It’s a journey that needs organisational changes and that needs process changes. It is a change journey in order to bring product success metrics to the table — not only have them on a dashboard and show them, but to really work with them. And you need to change the organisational process [to do this].”

Objectives and key results (OKR) come at the end of this journey, not the beginning.

“If you’re looking at product success, it starts with transparency and alignment,” Semmler said. “It starts with talking to your internal users, talking to your customers, talking to your audiences and to your business partners in order to understand what is driving their value.”

With this knowledge, an organisation can start looking at its audiences’ pain points and, through collaboration and cross-functional groups, to line up objectives.

“OKR as a thinking can really help us define product success metrics,” Semmler said.

He used John Doerr’s definition of OKR: a management methodology that helps ensure the company focuses efforts on the same important issues throughout the organisation.

Managing the journey

Semmler outlined the key steps of the product success journey:

• Objective: Where do we want to go? Who is our customer? What value will we create?
• Key result: How do we pace ourselves to see if we’re getting there (value indicators)? How do we measure if we’re creating value? What might be a good scale?
• Actions: What do we need to do to create value (value drivers)?

Looking at good success metrics for product, Semmler said it isn’t about milestones or tasks, but rather needs to be quantitative and measurable.

“We really need to look at leading indicators. How can I measure as early as possible, because product success is not just a tool. What you really want is to be able to change your organisations, change your processes, in order to create more value.”

It is a learning cycle, a continuous journey, he added. “You can learn, you can evaluate, and you can build more complex success metrics for even better product success.”

Why OKR is great thinking for measuring product success

Semmler shared the reasons objectives and key results are good metrics for product success.

• OKR forces an organisation to align and state clear objectives.
• It applies a holistic approach that looks at every dimension of goal attainment.
• It introduces value indicators to look at measurable results that show value and success on all levels.
• It understands measurable success requires collaborative learning and improvement, and cannot be achieved from day one.
• It makes success measurement a short-cycled approach.
• It reflects the environmental constraints and backgrounds of each organisation.

So what are Semmler's specific recommendations for product success metrics? It depends, he said.

“Each organisation is different. It depends on your organisational objectives and it depends on what journey you are on. You need to start simple in order to later get to more complex stuff.”

The good news is the product team in an organisation does not have to define product success on its own — and in fact, should not.

“Product success definition should not be the job of product managers,” Semmler said. “It’s our job to facilitate the discussion. It’s our job to bring everyone to the table in order to really look at what product success means.”

North Star metrics at the Financial Times

Lucy Butler, director of analytics at Financial Times (FT), outlined for INMA members the North Star metrics her organisation uses for product success.

She echoed Semmler in stressing that this is an ever-evolving journey — one that is leading FT towards a more comprehensive North Star. As the company’s business has grown, it has moved from a focus on simple, volume-based metrics to more sophisticated value-based metrics.

From an early focus in 2005-2014 on traffic, FT has moved through strategies built around quality visits and engagement to the current North Star metric of lifetime value (LTV).

“It’s really important to note that these metrics do not replace each other,” Butler said. “The complex metrics do not replace the simple metrics, they just build on them.”

Her team used RFV (recency, frequency, volume) as a leading indicator of acquisition and retention, looking at engagement, cancellation and conversion rates, and revenue versus usage. In addition, RFV is an important predictor of LTV.

LTV, in turn, provides a unified “common currency” for tricky comparisons, such as acquisition versus retention and B2B versus B2C in deciding where to put more focus and resources.

“It’s a really nice framework to be able to understand those trade-offs better,” she said.

Standardised frameworks

FT has built standardised methodologies for common scenarios that work with an opportunity assessment framework, Butler said. This is essentially a way of taking all the common types of opportunities and providing a consistent set of logic to them. This feeds off a set of common benchmarks, resulting in a template outpout to consistently compare opportunities.

“It enables us to have a ‘like-to-like’ comparison, and that’s been really helpful for us in getting everyone to speak the same language,” she said.

Simple metrics are still critical, she added, as the day-to-day levers that move the FT North Star.

“The more sophisticated metrics are composed of the less sophisticated metrics. There are certain metrics that are really helpful because they can move quite quickly on a day-to-day basis — or they may be much more direct in terms of the thing you are actually trying to move.”

These simple metrics then ladder up into the more complex ones that allow the organisation to measure for LTV.

“It helps us contextualise what those individual KPIs mean for us in terms of value,” Butler said.

A final caveat

“One thing that I think is really important is that we’re not actually trying to move the metric,” Butler cautioned. “We’re trying to achieve an outcome, and the metric is a proxy for that outcome.”

As such, they’re always a bit imperfect, she acknowledged. But those metrics as proxies help turn intangible outcomes (such as readers’ perceived value) into something more measurable (such as RFV).

“For example, with RFV, we’re not actually trying to move recency, frequency, and volume. What we’re trying to do is build customer loyalty and perceived value of our product and engagement with our product. We need to be able to measure it with something tangible.”

She stressed that it’s very important not to lose goal of the outcome that each metric represents — and to be aware of the fact that it’s possible to move a metric without actually moving the underlying outcome.

In the end, she advised to always keep sight of the ultimate goal: the outcome the metric represents.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .