If there is only one word to describe how FT Strategies wants publishers to think about experimentation, it’s iterative. 

Daisy Donald, principal at FT Strategies in the UK, calls experimentation their secret weapon: “We’ve really put experimentation at the heart of decision-making because those tiny gains can really add up to huge gains overall,” Donald said.

During Tuesday’s INMA Media Subscriptions Town Hall, three news media companies who worked with FT Strategies shared their experimentation success stories. Donald also shared one from her company, using My FT as an example. FT Strategies has a dedicated team that continues to make changes to the project, which has been around for years.  

“It’s a kind of active personalisation. The customer chooses the topics they want and they can follow that and we still now, years later, are making iterative improvements on that experience to try to get those marginal gains,” Donald said. “We know if we can get people to use My FT, they will stay with us longer and increase that lifetime value.” 

Tim Part, senior manager at FT Strategies, says there is magic in experimentation. The company believes experimenting and testing have many benefits and shared with INMA members the four they consider to be the most important.

Four reasons to experiment

1. Experimentation enables accelerated learning.

“Our process forces experiments to minimise the number of variables and test risks and not products,” Part said. “That helps you learn faster, learn better and do things in the right order.”

2. Experimenting saves money.

Many times, companies think they need certain tech to solve problems so they implement products and not solutions, Part said. The FT had the same problem, which really pushed the company toward experimentation. The FT spent a lot of money on a new recommendation system and found it wasn’t much better than them putting up a “most read” articles panel, Part said.

“If we had experimented properly, we wouldn’t perhaps have invested that much money in getting a system that essentially we didn’t need.”

He urges companies to experiment iteratively, saying it’s important to conduct smaller experiments where you aren’t committing as many people and other resources. This way you can learn quicker and stop if you need to. 

3. Experimentation promotes reader centricity.

“The process forces you to think about the benefits for the reader so you’re optimising for their needs rather than focusing on optimizing a product,” Part said. 

4. Experimentation builds a data-driven culture.

Part wants everyone in the organisation to share a common goal and go on the same journey together. 

“We think a business that experiments together, succeeds together and also fails together,” Part said. “But ultimately you’re becoming a learning organisation which is what experimentation is all about.”

4 stages of experimentation

1. Choose the right experiments.

It should prove or disprove an idea you have about growing your business. Part wants companies to weigh two variables. The first being impact. It’s important to see how much impact a potential change will have on the outcome you want to achieve. The other is risk. How difficult is it going to be to implement? 

“Those two things in combination will help you to understand in what order you should perform experiments,” Part said. “Generally you’d start with something high impact and low risk. That by definition is a quick win.”

Part says to first analyse your capabilities. You need to know what ability you have to experiment. Not only should you be thinking about what resources you have but consider whether it make sense to carry out certain experiments.

You’ll also need to build an action plan. Part recommends allocating your experiments in quarters over the next year to create a road map that people can follow. 

“Most importantly it will help make sure you are experimenting enough, a manageable amount, and over all the key areas you need to experiment in,” Part said. 

2. Lay the foundations.

Part says to plan rigorously before you start experimenting. There are three parts to doing this:

1. Define the context. Examine what you are trying to prove: “In that way, we can design the experiment in the best way possible and prove the risks and assumptions that are inherent in your hypothesis,” Part said. 

2. Do your research. Chances are you’ve experimented before in some of the areas where you’re experimenting in: “It’s important you work out what you’ve done before so you don't repeat your effort,” Part said.

3. Identify risks. “When you experiment, what you’re doing is not seeing if something actually works,” Part said. “What you’re doing is testing your risks to see whether you can proceed and scale up or you’re testing assumptions, testing things you don't know.”

3. Create a robust design.

Part says there are many types of experiments that work. You’ll want to select the best format for your company. 

“You don’t always have to just build the product and see it,” Part said. “You can start with a few phone calls to readers to see what they think. You can start with sketching things out and testing a design with people. There are iterative little ways that stop you from overinvesting in terms of time and money and test these risks and assumptions iteratively.” 

You also need to know what success looks like and how to gauge whether your hypothesis is proven or it’s time to move on to the next experiment. Part says to be sure to measure both primary and secondary metrics.  

“If your experiment, for example, is to prove conversion rate, are there other indicators that show whether you’re having an effect? Perhaps more people start the pay flow, but don't ultimately convert,” Part said. “That tells you something else about the process. It shows you new areas to experiment in.” 

4. Analyse your success and experiment further.  

“Experiments can be short, but it’s important to do follow up analysis,” Part said. 

You may not have metrics right away on something like retention. You want to plan out a follow-up to track retention over the next year. Part also says to look ahead to the next experiment and how to use the insights from your results to test further and in the right ways. He also encourages companies to share results. 

“Experiments shouldn’t be kept in house or kept in a small team,” Part said. “It’s important you share both successes and failures around the organisation.” 

Here are the next steps Part wants companies to take: 

1. Download FT Strategies’ report on the GNI Subscriptions Lab Europe 2021.

2. Evaluate your goals: Think about what it is you want to achieve: “The more ambitious you are, the more fun experimentation is and at the end, the better the results it’s going to give you,” Part said. 

3. Talk to your colleagues, involve everyone, and find good ideas on what do you want to test.

The benefit of this information is people understand the methodology, he said. When people buy into the successes, they get excited and want to help. This increases the pool of people you can have join your experimentation team.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .