Experimentation allows companies to gain incredible insights into product launches, but not all companies are taking advantage of this opportunity. 

“You can see the future,” said Peter Gray, vice president of product at Dow Jones. “Any idea [or] hypothesis that you have about an existing product, you have the ability to see the future and show it to a small percentage of your audience and find out what would happen if you did it for real.

“Experimentation is the best business tool that has ever existed.”

Many times, companies mistakenly believe experimentation opens the door to risk, but he said experimentation reduces risk rather than introducing it. He said he believes experimentation is a must, not a choice within a business. 

The experimentation mindset

Gray said teams must embody an experimentation mindset to become an experiment-driven brand. The first step is for leadership to change its mindset, as it sets the entire team up to change its mindset and processes. 

Failure is a part of the experimentation mindset as well: “It is also likely to fail, and that’s okay.”

User experience research is a tool that can refine and increase the success percentage before a launch. “When we have designs of different variations of something, we will show those to people before we launch it as a live experiment,” Gray said. 

He said he used to think this wasted time but has found the opposite to be true: “It reduces the number of experiments you have to run to get to the end result, and experiments are much more expensive than user research.” 

“We can all practise the experimentation mindset, regardless of whether we have access to experimentation functionality,” he said.

Structuring experiments

The first part of the experimentation process is gathering the objectives and key results (OKRs). 

Next is determining where to test these experiments. “We do a site for any metric we are trying to track, such as every 300 subscriptions, where do they come from?” Gray said. “It’s hard to make sure you’re going to maximise the ROI of your efforts if you don’t know where the organic action is.”

That is followed by forming a hypothesis based on analytics, past experiments, and UX research followed by a rigorous sequence test process. “With experimentation, you’re never arguing what to do, but what sequence to do it in.”

Finally, draft designs are created for the experiment and the experiment is launched. 

From there, the product either goes into implementation within the brand or to an insight repository. If the launch was not successful, it goes back to the drawing board to look for possible other ways to execute the experiment. 

“Giving your business focus through knowing what doesn’t work, is often just as valuable as knowing what does,” Gray said. 

Experimentation insights

Experiments with subscription offers provide “a shocking amount of value hiding in plain sight,” Gray said, and he showed examples of how publishers can leverage that information. 

With the launch of a member check-out page, The Wall Street Journal received feedback to eliminate the unnecessary graphics and copy telling the user things they already knew about the brand before getting to that page. 

Gray’s second example underscored the importance of “telling your customer what you thought they already knew.” 

In this case, a call to join the Wall Street Journal membership used two CTAs: one reiterating the credibility of the brand, and the second simply reminding the user what they were about to buy. 

“We thought of the dumbest and simplest thing in the moment, we thought we didn’t need to say it,” he said. “That had a 37% uplift on purchase, just by clarifying what was going on.”

Simply softening the call of action language helped Wall Street Journal gain new customers.  

Experimentation also was successful for its members’ newsletter page. Feedback suggested making the flow easier and straightforward by removing extra space, blocks, and colours, Gray said: “The eye is a muscle, you need to tell the eye where you’d like it to go.”

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .