Rubina Madan Fillion, associate editorial director of AI initiatives at The New York Times, has clear advice: You have to ensure humans are part of the process from early on, and you should make sure they are journalists. 

She directly asked about 100 reporters and editors how they would use AI, through focus groups and demonstrations and also engaged them through a “very active” Slack channel — all part of a pilot that ran for two months.

“We listen more than we talk,” she said. “Two-thirds of the requests and frustrations that we were hearing about had to do with summarisation.”

By summarisation, she does not necessarily mean bullet-point summaries but the many tasks an editor has to do before a story can be published or promoted, such as formulating an SEO headline or deck, blurbs for newsletters, or the last few minutes of a podcast where they say, “These are the news headlines you’re missing today and here is a very brief summary of other articles.” 

The NYT built a tool called Echo, which takes their articles and summarises them in any way a journalist wants. Its interface includes a list of articles they are looking at. Journalists can use preset prompts or write their own. Echo then takes the information from articles and writes it in another form.

Evaluation was tricky, Fillion said. How does one create quantitative metrics for writing, an activity that is inherently qualitative? 

She also found “the outputs are often quite mediocre because they were not trained on high-quality journalism.”

How could they get it closer to what the Times would actually have an editor write? And that proved to also be a difficult question to answer because even two editors sitting next to each other could have different opinions about what made a summary good. It was hard to articulate.

The team undertook an iterative process to understand quality. What were the editors’ requirements? They found many wanted to avoid jargon, acronyms and long sentences, which helped the machine understand what made a good summary.

But even asking for feedback was a delicate process. Fillion wanted to make sure it felt like editors were saving time, so she could not make any evaluation process too onerous. 

If you’d like to subscribe to my bi-weekly newsletter, INMA members can do so here.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .