News organisations are facing a crucial decision: optimise for AI search or block it. Inaction isn’t a neutral choice — it has consequences.

Whatever you decide, it needs to be measured. Without data, you’re making a blind decision. You need to know who is scraping your content, what they’re taking, and how frequently they’re doing it.

One option is to optimise for AI search when it comes to general news or Q&A content. Although there are papers published on optimising for AI search over regular SEO, there are still many unknowns.

Even the AI companies themselves cannot give hard guidance as it is down to the algorithm, which is highly complex and generative by nature. A single query could give many different results, unlike the search we have been used to for the past two decades. 

We’re already seeing changes to search and discovery. It’s virtually impossible to measure the affects of Google’s AIO as traffic is not discernible between the two through Google or other analytics tools. 

Even if you work out how to optimise for AI search, you are still relying on ranking above competitors and banking on the slim chance that consumers will actually click through. AI search is also known as an answer engine because generally they give people the answer they are looking for rather than a snippet of content that encourages consumers to clock through. 

Also, if you allow bots to scrape your site, your content could be used, sold, resold, and repurposed in places you have zero control over. If you’re hoping to secure a licensing deal with AI companies, ask yourself: Why would they pay for something they’re already getting for free?

Think you’re protected by robots.txt? Unfortunately, you’re not. Few companies actually abide by it — it’s a gentleman’s agreement at best. Take this example: I asked Perplexity if they follow robots.txt. They said yes. Then I asked if they use third-party data sources that may not follow the rules. They also said yes.

Personally, I think optimising for AI search is a hard case to make. If you think differently, I’d genuinely love to hear from you. It’s a complex area and we need to look at all possible sides. 

Of course, not all AI companies scrape without permission or rely on third-party data. OpenAI, for example, has been clear about following the rules. Notably, they’re also one of the few AI companies making licensing deals with publishers — recognising the value of content.

Blocking access to your most valuable content may bring companies to the negotiating table.

You might argue all of your content is valuable — and I wouldn’t disagree. But some content holds more value than others. Do you know which content that is? Because that’s your key negotiating point with AI companies.

And here’s the best part: You can measure this for free. Both ScalePost and Tollbit offer free tools to help you track scraping activity and even monetise your content. These tools are free because they generate revenue further down the pipeline through content licensing — a shared goal for many reading this.

There may be other companies out there I haven’t yet come across, and your cybersecurity provider may already partner with one of them.

Bottom line: As I continue analysing the role of news in the AI-driven landscape, one thing is crystal clear: You need to either block the bots or measure them. Now. I urge you to work with third-party companies dedicated to detecting and managing bot activity. Measure your content, identify where your true value lies, and negotiate from a position of strength.

If you’d like to subscribe to my bi-weekly newsletter, INMA members can do so here.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .