Many of the 125-ish INMA members whom I have heard from or spoken to over the past nine months are working on creating chat products.

In fact, we had 188 participants registered for a Webinar on chat products on September 4, so we asked them a couple of questions to get some idea of what people were thinking. (Feel free to watch the replay — come for the chatbots, stay for the Beethoven and the Lord of the Rings reference.)

The first was whether they have a chat product in the works already (admittedly, not the best question since presumably all viewers of a Webinar on chat products would be at least somewhat interested in the topic). A quarter of the respondents had already built one, one-fifth had one in the pipeline, while about 40% were considering creating one.

We also asked what they were worried about.

It looks like the biggest concerns over chat products centre on accuracy and profitability, which are concerns that we have discussed here before. But it was also interesting to see that some respondents flagged utility as a question mark: Is this actually a useful product that serves a need that the user has?

In fact, rapid prototyping — allowing you to see if the product solves the problem you think you have, learn quickly and fail quickly if needed — has emerged as one of the best practices around building a chat product. 

Here are some other useful insights from our two guests, The Washington Post Senior Editor for AI Strategy and Innovation Phoebe Connelly and Aftonbladet Deputy Managing Editor Martin Schori:

• Scope it narrowly. Ideally, on a topic that is not frequently and rapidly overtaken by events so the information it provides remains current.

• Be transparent. Tell your audience what you are thinking and doing. Both Aftonbladet and The Washington Post link their bots to a page that explains how the bot comes up with answers and why they built it.

• Train it on your own content for accuracy and use a RAG model. The object is to provide the curious news consumer with an entry point into a subject you have covered with both breadth and depth, and to help them quickly find what is relevant to them.

• Many users understand AI can make mistakes. Make it clear this is experimental technology and encourage readers to read the content that it links to.

• Users don’t mind AI. “If you ask them, they will say, oh, I don’t want anything that is AI-generated,” Schori said. But: “As long as the information provided is relevant and adds some value, people don’t really care.”

• Teach the bot to say it does not have the answer rather than letting it hallucinate. This does not come naturally to us in the news industry because we are uncomfortable telling readers we do not always have the answers.

• Use the questions readers are asking as data to shape your coverage of the topic. It is a rare glimpse inside their mind to see what is truly relevant to them on a subject.

• Offer the reader some prompts or sample questions to get them started, in addition to offering them a space to type in their own questions. Offer further prompts once those questions have been answered to keep them engaged and to keep showing them value.

• It does take time and work to build a high-quality chat product. (ICYMI: Mike Pletch from Canada’s Globe and Mail delivered a detailed master class session on this in May.) “Just because it’s published doesn’t mean it’s done. A lot of time needs to be spent checking all the questions, going through audience feedback, and adding more data,” as Schori said. “I think it was more work than we thought.”

• Use it to gather first-party data. You can use the chat product to drive registrations. Aftonbladet saw a 40% conversion rate.

• Build an LLM-neutral system rather than being rigidly locked into one.

• Put a daily ceiling on the number of questions that a user can ask if you are worried about cost. Also: Aftonbladet’s bot cost about €1,100 to run over a month.

• The most consistent feedback The Washington Post got was that readers loved the short answers. “Our audience is always asking for brevity,” Connelly said.

• The biggest challenge? Dealing with humans — getting stakeholder buy-in and answering broader questions about the role of AI in news rather than specifically about the product.

If you’d like to subscribe to my bi-weekly newsletter, INMA members can do so here.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .