The last few posts of mine have talked about various tips, techniques, and traps of using your data — be it “Big” or just plain old regular-sized data. This month, I thought I would toss into the conversation a topic that brings smiles or cringes, depending on your take on the subject. 

Recommendation engines.

Yep, I will go there. I hope we can navigate the topic and still sit down together and break bread someday.

Used properly, recommendation engines can encourage readers to dive deeper into content.
Used properly, recommendation engines can encourage readers to dive deeper into content.

So, what exactly is a recommendation engine? Think of it as the computerised version of a personal shopper. Like a personal shopper, the engine learns what you like and picks similar, or related, things for you to buy (or read).

How? With a complex algorithm that looks for subtle connections between past searches and outcomes. It looks at what you searched (or read) and finds people who searched, bought, or read the same in the past. Then it shows you what is either similar or what others bought (or read) as a suggestion for you.

For example, if you look for a fish tank, the engine knows that when other people looked at or bought a fish tank, they also looked for or bought aquarium filters. So the recommendation says “Hey Greg, you might want to buy aquarium filters to go with that tank.”

Done right, it is great. Done wrong (usually because a cookie dropped on your computer isn’t expired correctly), the result ends up on the 11:00 news or goes viral on social networks. We have all heard the story of the dad finding out his daughter was pregnant by one of these recommendations gone wrong.


If you want to kick the tires on an engine, play around on the Apple music store or the shop Amazon. (Hey, this could be the good excuse you need to shop at work right?) Search for a few items, then scroll down the screen. You will see a section showing “Your recently viewed items and featured recommendations.”

This is the recommendation engine at work. If you are a Prime member, the section starts to get spooky — but it works, usually. (As a sidenote to Amazon: For your information, I bought A French Kiss with Death a long time ago. But not from you, so please quit trying to sell me a copy.)

In the context of squished-tree newspapering, the editor was the recommendation engine. We had 32 broadsheet pages to fill. The editor(s) selected the best stories/photos to fill the 32 pages, and we were all happy.

Today, in the context of digital news delivery, the editor performs the same function. But rather than fitting into a fixed number of pages, the effort is to fit within the site’s page design parameters.

Today’s challenge, though, is that we all know that for all intents and purposes, the Internet is infinite. The editors don’t have to limit the site to the design constraints. They could post everything. Isn’t that what the search engines do? Anyway, rather than posting everything, news sites do constrain the content to build a high-value product that is scaled to fit the audience.

Recommendation engines are a very clever way to extend a site without having to directly post a zillion stories.

Editors have a variety of sources for content to post, and, by knowing their audience and place of honour in the community, they select the best content and dismiss the rest. With a little bit of page re-design, and a good recommendation engine, the dismissed stories can be brought in through a unique feed to each reader.

Without the engine, we all do this with “more stories” teasers all over our sites. The engine gives you a targeted and smarter way to engage with the reader. Rather than a blanket “more stories” nudge, with a little page design work, sections of each page could receive a treatment that says “recommended for you,” or a more subtle “our picks.”

In either case, the recommendation engine would fill the section.

In the end, whether on squished trees or glass, the exercise is to get the (massive) content available down to a manageable summary and to place each story appropriately on our branded products and/or sites. The digital news medium really is not infinite; it is still a balancing act within a designed delivery vehicle.

The editing work is that of the recommendation engine — human or machine. Digital has an elegant twist. In digital with page design work, extra stories can slide into the page design on the fly.

The recommendation engine gives editors another tool to build connections with readers and keep them on their own sites. Anyone with an Internet connection can override the editor and search away to get additional content delivered.

We are all painfully aware that once a customer clicks/touches away from your site, it is hard to get them back. The recommendation engine needs to be considered as a tool to extend relevance rather than a loss of control.

Used correctly, the recommendation engine gives the editor another tool in the balancing act of content selection. You can continue to build your pages as you always have, or build them with a gateway to unlimited length with a simple change to your page design.

This technique keeps your page length short (i.e. the critical quick load time), but introduces a single-click branch where the reader can extend the content viewed as they want.

You have to decide if you want to get into the computerised recommendation business. I argue that the decision has already been made. People are clicking out of your site after a story or two. Why not give them an opportunity to stay?

The computerised recommendation engine is the new tool you need to implement to keep people on your site by analysing the content consumed and offering more.

Assuming you are with me and believe yes, let’s do this, then the next question is: Do you build or buy the tool?

With the Big Data databases, get click-stream level data coming off the Web site(s) that many publishers have available — the data waiting for the engines to mine and leverage.

Should you build the engine? Well, with the right level of Python, R, statistical, and API development/deployment skills and resources, you could build your own engine. The tricky part is moving from “I know” to “you would like.”

You need to have the reserve content to offer them. You have to have the tagging and bot design skills to do the internal and external content indexing to bring back a link/image to the recommended content section. It gets complex to pull it all together when recommending content beyond your CMS library.

Building isn’t for the faint of heart. For those wanting to dive deeper, there is some nice detail on the problems with building your own on Data Community DC.

As for the buy option, there are many commercial recommendation engines available, to the point where there is a Web site that tracks what they call the top 33 engines ( The engines are e-commerce focused, but it is a starting point for discussion to move into a content-selection rather than a purchase-decision recommendation.

I’m sure Google and Amazon would be glad to help you as well. All of these vendors and providers ultimately can do the heavy algorithm lifting, support of the technology, and implementation expertise as they have a wide range of customers. This breadth gives them a depth of experience and technical advantage that is hard to overcome in the build-your-own case.

So, where do you fall? Add a section on the pages for computerised recommendations or continue with a fixed-length Web site? Are you willing to let a computer algorithm drive the content options presented on a section of the Web site? Is there a happy medium?

Will you implement in an overt fashion or hide it with a softer heading? Do it via a section or mask the recommendations with stealth/dynamic page load? The questions here are to help you in your internal discussions.