Quality journalism is massively under-valuated, with no advertising correlation between production costs and economic value — and editorial quality production often gets buried amid new products. In a live INMA Webinar on Wednesday, famed French media expert Frederic Filloux outlined the algorithms he and his team are developing to define quality, quantify journalism, surface stories, and score them — at scale and with machine-readable signals.

Filloux outlined the work he’s done on this topic at Stanford, discussing the two major problems he believes media organisations need to solve right now:

• The massive undervaluation of quality journalism.
• The untapped potential for product.

First, we must define quality, which Filloux describes as the amount of resources deployed by a media organisation to cover the news, in terms of:

• Value-added.
• Workforce.
• Talent.
• Expertise.
• Innovation.
• Ethics/transparency.

Scoring for quality

Filloux then outlined how NQS (News Quality Sourcing) works to address untapped potential. The publisher presents its stories to the NQS platform, which then assesses the content to come up with a global score for quality. This process would be done at scale, automatically, and using ML algorithms.

“A reliable score makes anything possible,” Filloux said. When it comes to advertising currently, the problem is advertisers pay the same price, regardless of content quality. “The way I see that a platform could improve this is by scoring the news for quality in this way.”

The ad server would then be able to differentiate between premium ads (at a higher cost) and non-premium (lower cost). “This would enable them to serve the highest value advertising based on the score,” Filloux said.

Why should advertising reward quality? “Indicators are blinking red in terms of revenue, fraud, viewability, and adblocking,” Filloux said. “They currently do this by looking at the ‘average’ — however, the problem with this is that average means nothing.”

Brand safety is a major issue, he added. Great news content calls for high demographics — more affluent, educated, and better engaged. Quality content is also better for innovative ad formats.

Filloux also cautioned that ROI is not everything, referencing a quote from Digiday that he recently read: “We do programmatic because we need to be where our consumers are. But the programmatic buys we’re making now prioritise quality.”

Recommendation engines

Recommendation engines are the easiest way to get readers’ attention, Filloux said; but the problem is they “sucks terribly.” There is little to no relevancy in the stories that are served, and there are lots of false-positives that show a relationship where none exists. Many publishers also delegate this to outsourced platforms. “There is no signal for quality of the story.“

The NQS platform would improve this by vetting the content. A publisher’s archives would go through NQS, which would lift the most qualitative stories from the archives. This would result in value-added content instead of an uncertain reliance to keywords. The model’s dials would be tuned for better relevancy, resulting in increased page views, session duration, and premium subscriptions.

“This could be a great engine for premium product,” Filloux said.

This would prevent what he calls the “zombie stories” syndrome — defined as stories that fall in the depth of a Web site way before achieving their full potential of viewership. “If we can find a way to lift stories from the archives based on their quality and relevance, it can definitely have an impact on readership numbers.”

Personalisation

Publishers are sharing the same set of metadata for each story. The idea of the NQS Quality Tag is to include all the reader’s preferences, such as topics and types of preferred articles. “The whole idea is to have stories from the platform, shared with the same syntax and quality score provided by NQS, which would be used to prioritise the stories,” Filloux said.

This would provide:

• Real-time extraction of value added links.
• Maximum relevance.
• Ability to package in premium products (into a newsletter, for instance).

“This would take advantage of the expertise of the newsroom and pass that along to the reader,” Filloux said, outlining several upsides across the board for using NQS:

• New product opportunities.
• Increase of the “destination” dimension of news sites.
• No more leakage to other players, such as aggregators and curators.
• More time spent, deeper engagement, and more loyalty.
• More revenue.

So, how do we build it? Filloux said there are three approaches to building the Quality Tag system.

1. Feature-based: This approach relies on signals. There are quantitative signals that indicate the quality of the content, such as bylines, photo and video, data richness, link density, and advertising density. Text analysis includes word count, quotes analysis, Flesch reading ease, etc.

There is also another set of subjective signals, including writing style, story type, thoroughness, and timeliness, which can only be analysed by humans. This is something that NQS is building for the future.

2. Deep learning-based (text): This approach uses natural language processing with a mathematical-probability oriented approach. Using a “text embedding” technique (vectorisation), it is processed by a neural network.

“It basically transforms text into a bunch of numbers, from which we can find hidden patterns,” Filloux said. “From this, we can deduct a score probability for the value of the article.”

3. Deep learning-based (computer vision). This is very straightforward, yet no one has every tried it. The computer would take the content to build classifiers of “good” versus “bad” Web sites, using the computer vision algorithms, to detect graphic patterns.

“The whole idea is to use these three approaches stacked upon each other,” Filloux said.

The advantage of this is better performance, while the drawback is that it’s less transparent.

Filloux closed his presentation by sharing two links where INMA members can find more information: Newsqualityscoring.com, which will be rebranded to Deepnews.ai within a week.

Q&A

INMA: Is this a paid service, and when is it available?

Filloux: Right now we don’t have a beta version running; we are still working on the model. We have come up with the list of indicators, and we know that these indicators make sense and we’re on the right track.

INMA: Do you plan on having it work in other languages besides English?

Filloux: There is no obstacle to that. When it comes to the deep-learning approach, those models have already been converted to other languages. If we are able to find partners, then we can do it.

INMA: Is there a danger for readers who aren’t able to afford quality news, and how will we be able to keep influencers from reaching a more narrow group?

Filloux: I would love for this to be able to capture the small blog [as well as major media companies]. In an ideal world, I would love to see the news quality scoring reach even the small, relatively unknown sites and publications. While this will be a paid service, I intend to have a free version that will be available to small Web sites.

INMA: How do you go about selecting human analysers?

Filloux: We have two choices: relying on putting the job on platforms or go directly to qualified professionals in the media industry.

INMA: Given how fast the news cycle is working, where do you see the human intervention part fitting in?

Filloux: This model is aimed at interfacing with stories that are not necessarily breaking stories. Long-form, feature stories, archives, etc.

INMA: Have you had conversations with Facebook regarding qualifying news, the Trust project, Google fact-checking, etc. Will people like this be part of your conversation?

Filloux: Yes, we are partners with The Trust Project, Google Digital News Initiative, and others. Quality is long-form; it does not fit the Facebook model, which is based on churn, clicks, likes, etc. They are not interested in quality and value-added content in the News Feed.

INMA: How will you stack for bias?

Filloux: This is a key aspect for all models. Bias is the main problem for AI models; they are basically made by white males, which triggers a lot of problems. The interface will run thousands of stories, and I think we should run the whole batch of this through the human scoring interface to see what the correlation is.

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .