Hi there.

If you’re trying to decide whether to optimise for AI search or block the AI bots, today’s newsletter is for you. I’ve had this discussion with countless news organisations. After each conversation, the case to optimise becomes narrower in my humble opinion. 

There is also an avalanche of tech updates I wanted to give you a quick rundown about. Each could be an article in itself, but instead of writing a book (!) I want to share key updates and why they matter for news. 

If you’re really looking at Big Tech, how AI is changing the entire ecosystem of information, and how news orgs are using AI, join us in the heart of where it’s happening: Silicon Valley. We’re gathering the people making this new future for our all new Media, Tech, and AI event this October. I hope you can join us. 

If there is anything I can help with, any views or debate you want to share, please drop me a line. I’m at Jodie.hopperton@INMA.org.

Thanks, Jodie

To block the bots or optimise for AI search?

News organisations are facing a crucial decision: optimise for AI search or block it. Inaction isn’t a neutral choice — it has consequences. Whatever you decide, it needs to be measured. Without data, you’re making a blind decision. You need to know who is scraping your content, what they’re taking, and how frequently they’re doing it.

One option is to optimise for AI search when it comes to general news or Q&A content. Although there are papers published on optimising for AI search over regular SEO, there are still many unknowns. Even the AI companies themselves cannot give hard guidance as it is down to the algorithm, which is highly complex and generative by nature. A single query could give many different results, unlike the search we have been used to for the past two decades. 

We’re already seeing changes to search and discovery. It’s virtually impossible to measure the affects of Google’s AIO as traffic is not discernible between the two through Google or other analytics tools. 

Even if you work out how to optimise for AI search, you are still relying on ranking above competitors and banking on the slim chance that consumers will actually click through. AI search is also known as an answer engine because generally they give people the answer they are looking for rather than a snippet of content that encourages consumers to clock through. 

Also, if you allow bots to scrape your site, your content could be used, sold, resold, and repurposed in places you have zero control over. If you’re hoping to secure a licensing deal with AI companies, ask yourself: Why would they pay for something they’re already getting for free?

Think you’re protected by robots.txt? Unfortunately, you’re not. Few companies actually abide by it — it’s a gentleman’s agreement at best. Take this example: I asked Perplexity if they follow robots.txt. They said yes. Then I asked if they use third-party data sources that may not follow the rules. They also said yes.

Personally, I think optimising for AI search is a hard case to make. If you think differently, I’d genuinely love to hear from you. It’s a complex area and we need to look at all possible sides. 

Of course, not all AI companies scrape without permission or rely on third-party data. OpenAI, for example, has been clear about following the rules. Notably, they’re also one of the few AI companies making licensing deals with publishers — recognising the value of content.

Blocking access to your most valuable content may bring companies to the negotiating table.

You might argue all of your content is valuable — and I wouldn’t disagree. But some content holds more value than others. Do you know which content that is? Because that’s your key negotiating point with AI companies.

And here’s the best part: You can measure this for free. Both ScalePost and Tollbit offer free tools to help you track scraping activity and even monetise your content. These tools are free because they generate revenue further down the pipeline through content licensing — a shared goal for many reading this.

There may be other companies out there I haven’t yet come across, and your cybersecurity provider may already partner with one of them.

Bottom line: As I continue analysing the role of news in the AI-driven landscape, one thing is crystal clear: You need to either block the bots or measure them. Now. I urge you to work with third-party companies dedicated to detecting and managing bot activity. Measure your content, identify where your true value lies, and negotiate from a position of strength.

Date for the diary: March 18, 20, 25: Building Direct Audiences Master Class

We’re moving beyonds a traffic era with renewed impetus for direct, loyal audiences. But times are changing as search and social are in flux and AI is changing the information ecosystem. In this master class, we’re looking at new ways to attract readers, how to create daily routines that encourage habit, and how to build communities. 

We have a stellar faculty, including Jonathan Knight, head of NYTimes Games; Amanda Rottier, head of CNN’s Growth; and Liesbeth Nizet, who is focused on monetising future audiences for one of Europe’s biggest media organisations, Mediahuis.

Check out the full event agenda and faculty here.

Big Tech updates and why they matter for news

Over the last few weeks, there have been a few announcements. Some splashy, some under the radar. All with potential impact for news. Today, I share three I think are worth paying attention to, all for different reasons.

1. OpenAI’s reasoning tools show their workings

Named O1 and O3, there is a professional level tool that can reason out complex questions and a slightly more lightweight version that is available to pro users. These are less for question and answer and more to solve strategic questions. 

Check out this example, which could have helped me plan the best time for our upcoming Media, Tech, and AI event in San Francisco (thankfully we chose dates this AI approves of).

What if consumers could use this to help follow reasoning and form their own opinions, seeing which facts they want to switch out or disagree with? Imagine asking, “Which candidate should I vote for?” and seeing everything the system knows about you (a lot, see here) and the various policies.

2. Apple News is getting a “+ Food”

On February 21, Apple announced that “Apple News+ subscribers will soon have access to tens of thousands of recipes, restaurant reviews, kitchen tips, and more, right in the Apple News app.”

As always with Apple, the design is beautiful. And it seems it is being rolled into the US$12.99/month price point. Compare this to NYTimes US$6/month for cooking and US$25 for their all-in subscription. Apple News is currently only in four countries, but in January this year the FT reported “Apple steps up expansion of its News platform.”

While some news organisations report meaningful traffic from Apple, it’s hard not to consider cannibalisation, especially with such an attractive consumer price point. 

3. Grok is actually pretty good

I’ll admit I had preemptively dismissed Grok by X. I have not deleted the X app (and still mostly call it Twitter, but I barely look at it anymore). Then someone messaged me, “Playing with Grok on X. News summary is perfect.” 

This is the summary they sent me:

When I played around with it, I couldn’t replicate the bulleted list. However, I did get an impressive round-up of news. It was more wordy and article-like, as you see below. 

The sourcing was a series of posts on X in a swipeable format plus a clickable box of “relevant Web pages.” Interestingly, the second listing on the Web pages was clearly copyrighted to a media company yet taken from a syndicated site, which shows how convoluted AI licensing can get.

Grok also has advanced capabilities in the same way as OpenAI above so I ran the same query. Grok told me I was wrong — for valid reasons it seems. Again, it’s interesting to see the reasoning. Grok gives you the choice to see it as a bar entitled “thinking for X seconds.” And although a search takes longer — I think this was 37 seconds — the reasoning is excellent.

I should not have been so quick to dismiss Grok. And it also shows there are more LLMs being launched. At the end of 2024, there were four clear winners, as per my report. Grok shows this is up for grabs.

4. Amazon announced its Alexa+

Perhaps the most interesting of all the announcements, at least from a news media perspective, was the most low key: Amazon’s Alexa+. In their words: “It’s not just about knowing a lot of things, it’s about deeply understanding and bringing it all together into an accurate and real-time response.”

Why is this so interesting? 

There are already 600 million Echo devices. These are in people’s homes just waiting for this software update. The voice seems to be as good as OpenAI, and it can also run agentic tasks and seems to plug seamlessly into smarthome set-ups — which means it can actually do useful things right now.

They may be behind the AI race, but it seems Amazon is taking it seriously and has really shown up. 

As with all good AI, it’s highly personalised. As they say: “The new Alexa is highly personalised — and gives you opportunities to personalise further. She knows what you’ve bought, what you’ve listened to, the videos you’ve watched, the address you ship things to, and how you like to pay — but you can also ask her to remember things that will make the experience more useful for you. You can tell her things like family recipes, important dates, facts, dietary preferences, and more — and she can apply that knowledge to take useful action.”

How does it relate to news? 

According to Axios here, Amazon has been in discussion with news media. When I reached out to the team at Amazon, I didn’t get a response. If you’ve been in talks with them, I’d love to hear from you confidentially. And I’ll do more digging over the coming weeks and report back.  

About this newsletter 

Today’s newsletter is written by Jodie Hopperton, based in Los Angeles and lead for the INMA Product and Tech Initiative. Jodie will share research, case studies, and thought leadership on the topic of global news media product.

This newsletter is a public face of the Product and Tech Initiative by INMA, outlined here. E-mail Jodie at jodie.hopperton@inma.org with thoughts, suggestions, and questions. Sign up to our Slack channel.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .