While A/B testing best practices are widely documented, the difference between incremental gains and meaningful growth lies in how those practices are applied.

Advanced experimentation is not about running more tests. It’s about running smarter, more intentional tests grounded in user behaviour, clear hypotheses, and business outcomes. It requires connecting user experience (UX) decisions to value perception, aligning metrics to specific goals, and designing experiments that can be confidently scaled.

The Tiny Tiles paywall test run by The Washington Post illustrates this difference in practice: moving beyond surface-level optimisation to a disciplined, insight-driven approach that directly influences subscription growth and revenue.

Designing experiments that drive growth

Start with a clear hypothesis rooted in best practices

The Tiny Tiles test hypothesis was formed based on UX best practices as well as previous learnings from other acquisition testing at The Post.

Our hypothesis was that a shorter wall design, more prominent pricing, and clearer product benefits would increase the subscription and registration take rate. While Tiny Tiles introduced a new visual layout, its impact came from deeper changes, including:

• Elevating product benefits.
• Making pricing format larger and easier to understand.
• Reducing cognitive friction through a simplified design.

The new design was not just an aesthetic improvement. It also allowed for stronger perceived value at the point of conversion, emphasising what influences user decision-making through interface design.

Maintain clarity in experimentation and analysis

The Tiny Tiles test followed A/B testing principles, allowing the team to isolate what worked on each touchpoint. This included:

• Clear and consistent measurement, comparing the performance of the new design directly against the control or current design.
• Analysis across multiple audience segments and touchpoints.
• Clearly defined KPIs and statistically significant results.

Measure what drives business impact by test goal

The team determined clear and measurable KPIs that drive growth for two high-impact touchpoints — the paywall and registration wall — each of which has different goals.

The goal for the paywall was focused on subscription conversion and revenue. The KPIs we measured were:

• Subscription conversion rate (CVR).
• Product and billing cycle mix (monthly versus annual subscriptions).
• One-year and three-year revenue impact through customer lifetime value (CLV).

The goal for the registration wall was to turn anonymous users into known users through account creation while providing the option for subscription. The KPIs we measured were:

• Registration take rate.
• Subscription CVR.
• Product and billing cycle mix (monthly versus annual subscriptions).
• One-year and three-year revenue impact through CLV.

Base decisions in overall growth and iterate

The results from the test were clear: Paywall CVR increased with little shift in billing cycle mix makeup, resulting in an increase in both one-year and three-year revenue. The hypothesis was confirmed, and the new design was rolled out to 100% of traffic to maximise conversions and revenue.

Registration take rate and subscription CVR declined, resulting in a decline in one- and three-year revenue. The control was retained on 100% of traffic to protect and maximise registrations, conversions, and revenue.

The clear data and analysis allowed for tailored, confident scaling of the new Tiny Tiles design on the paywall while retaining the control design on the registration wall to maximise revenue and drive sustainable subscription growth.

The learnings from the registration wall results also highlight the importance of varied KPIs based on the unique desired outcome. It allowed for further hypotheses to be made around registration optimisation, which have informed future design principles and test ideas for continued learning and revenue growth.

Framework: from experiments to a sustainable growth engine 

The Tiny Tiles test reinforces a critical point: Advanced A/B testing is not just about identifying winners but rather understanding why something works, where it works, and where it does not.

A system that drives continuous learning, enables precise scaling, and ultimately builds a more effective and sustainable subscriber acquisition engine is what distinguishes advanced experimentation.

Use this framework in your organisation to rev up your growth engine:

1. Define the growth objective based on business impact.

2. Identify the high-impact touchpoints.

3. Form a hypothesis grounded in insight. Build your “if we change X, then Y will improve, because Z” hypotheses using UX best practices, past test learnings, and user behaviour data.

The Tiny Tiles example demonstrated shorter design plus clearer pricing plus stronger benefits led to a higher take rate due to improved value perception.

4. Design for value perception (not just user interface changes). Focus on what actually drives conversion: Elevate product benefits, simplify and clarify pricing, reduce cognitive friction, and make the value obvious at the decision moment.

5. Ensure results are actionable and trustworthy. Test against a clear control, isolate primary variables, measure consistently across variants, analyse across key audience segments, and require statistical significance before acting.

6. Define KPIs based on the specific goal and touchpoint.

7. Measure full business impact, and not just immediate lift. Look at the revenue impact (short- and long-term), changes in product mix, and the quality of users acquired.

8. Make decisions based on total growth impact. Roll out winning variants where they drive the primary KPI, retain control where performance declines, and tailor implementations to maximise impact.

9. Apply learnings to the next set of tests. Every result, positive or negative, fuels future growth.

To take full advantage of what you’ve learned, identify why a test succeeded or failed and develop new hypotheses from insights. Continuously refine design and messaging principles.

10. Scale what works, systematically. Roll out proven experiences across relevant surfaces and extend learnings into other transferable acquisition channels. Build repeatable testing pipelines.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .