Nearly every major news Web site relies on A/B testing for headlines. Editors often struggle to come up with additional headlines for their articles. So, how can AI be of assistance in headline testing tools?

What follows is the result of a collaboration between an editorial analytics company (smartocto) and a Dutch regional broadcaster (Omroep Brabant) to see if ChatGPT is able to generate compelling alternative headlines and whether this can be integrated into the workflow of a busy news site.

We discovered that ChatGPT often comes up with winning headlines, though there were practical challenges in using this approach. Ultimately, though, the final decision needs to rest with the editor.

The study: Compare the original with the winner

First, we developed a baseline. To establish that, smartocto analysed nearly 9,000 articles from 57 news sites around the world published during June 2023 that had undergone conventional, editor-led A/B headline testing.

The best metric to measure impact is the click-through rate (CTR), representing the percentage of Web site visitors who clicked on the headline. (The original headline might also be the winning headline, because we’re assuming the original headline would have been the headline if there hadn’t been a test. When calculating “winners,” the percentage of “loyalty clicks” was taken into account via the percentage of visitors who stayed on the page for 10 seconds or more, thus identifying clickbait.)

On average, the winning headline received 19.3% more clicks than the original headline.

ChatGPT results

Regarding the first results of the ChatGPT test with Omroep Brabant, of the 40 headlines tested, ChatGPT came up with the winner 23 times. The original headline won 17 times.

But it’s when we look at uplift that the result becomes interesting: Our findings show that, while it’s worth using ChatGPT, it’s better to come up with alternative headlines yourself. There’s a 19.3% uplift for the human editors and 9.2% increase in clicks for the ChatGPT option.

There’s potential for further study on this, but there are three key takeaways so far:

1. ChatGPT offers suggestions, not solutions.

Ultimately, ChatGPT is a suggestion machine, not a solutions machine. This gets to the core of what it (or any other AI tool) does. That’s why we had ChatGPT generate three alternative headlines, from which the editor would choose one.

In many cases, these headlines needed further adjustment for the following reasons:

• Suggestions were factually incorrect.
• Headlines provided were too clickbait-oriented.
• They contained too many emotional words like “heartbreaking” or “apocalyptic.”
• Headlines didn’t align with the Omroep Brabant brand and tone of voice.
• ChatGPT wouldn’t give suggestions on sensitive content such as rape or murder. The company behind ChatGPT, Open AI, thinks that’s dangerous.

2. The editor is in charge.

To be clear, the goal here wasn’t to determine whether ChatGPT can outperform journalists in headline creation, but rather to see if ChatGPT can be beneficial in helping to make suggestions.

One good example is the following. The original headline was: “Flames shoot out from the roof in a major house fire in Breda.” The alternatives were not compliant, but ChatGPT suggested adding that two houses were evacuated. The editor combined headlines to create the following: “Flames shoot out from the roof in a major house fire, two houses evacuated.” That headline won the test with a CTR of 11.35% against 8.46%.

Use ChatGPT as a sparring partner, much like you would with a colleague. “We were quite critical of the suggestions that ChatGPT came up with, but, often, we still saw a new perspective that we hadn’t thought of ourselves,” said Omroep Brabant editor Janneke Bosch.

3. Editors must pay attention to the ChatGPT prompt input.

As an editor, you can also modify the prompt, which aligns with your responsibility and journalistic ethics. If you want shorter headlines as suggestions, specify this. It’s easy to make a mistake here. During the first 20 articles tested, the prompt was very simple: “Create three alternative headlines for this article: [article URL].”

The problem was that ChatGPT 3.5 (the free version used during this study) cannot browse the Internet, so any alternative suggestions were solely based on the provided URL. Surprisingly, the input was still reasonably good. After that, smartocto’s prompt engineer helped and, together with Omroep Brabant, they have created a customised prompt.

In conclusion

Editors are generally great at crafting alternative headlines, but if the editorial team lacks the time or energy for brainstorming alternatives, ChatGPT can be a valuable collaborator with just a few clicks.

That said, it’s far from perfect. The editors at Omroep Brabant noticed ChatGPT often provided unusable suggestions when they wanted to express something unique in the headline — maybe to emphasise that someone has been interviewed or that it was a follow-up.

As with so much in the world of Big Data, the answers given are only ever as good as the questions asked. Whether or not editors feel that modifying prompts is more work than just generating alternatives through AI-free brainstorming is perhaps a personal preference, but there are A/B testing tools that can automate this task.

It’s a useful addition to editors’ toolkits for sure, but only if it’s utilised properly.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .