What is a recommender system?

A recommender system is a broad term for the infrastructure providing a personalized recommendation based on input data. Any online service that makes personalized recommendations has an underlying recommender system. Netflix, Pandora, Amazon, and YouTube, to name a few, all use recommender systems to suggest the most relevant content and purchases to their customers. Recommender systems are a major revenue driver, offering the personalization that plays a vital role in any customer-centric business.

The word personalized is important here, as a website can make a recommendation to the user without having a recommender system. For example, Wikipedia will always recommend the same articles for further reading at the bottom of the page no matter the identity of the user or their browsing history. This contrasts with sites that use recommender systems, such as Netflix or Amazon, where the recommended content is unique and personalized for each user.

Content-based vs. collaborative recommenders

There are two ways that recommender systems can approach the data to make their prediction: Content-based or collaborative recommenders.

1. Content-based recommenders

Content-based recommenders evaluate the user’s past behavior and the content items themselves to make recommendations. Pandora is an example of a content-based recommender. Their recommender system evaluates the user’s preferences and song characteristics like length, instruments, harmonies, genre, etc. It uses all this information to recommend new songs for that specific user.

Content-based recommenders evaluate the user’s past behavior and the content items themselves to make recommendations. It works with data that the user provides, either explicitly (rating) or implicitly (showing interest by clicking a link). With this data, a user profile is created, which helps the system provide more personalized suggestions to the user. The engine becomes more accurate as the user provides more inputs or takes actions on the recommendations.

Pandora is an example of a content-based recommender. Their recommender system evaluates the user’s preferences and song characteristics like length, instruments, harmonies, genre, etc. It uses all this information to recommend new songs for that specific user.

Image source: Business Insider

2. Collaborative recommenders

Collaborative recommenders, however, compare all users and user-item interactions (liked, disliked, clicked) in order to make recommendations. Generally, it is the process of filtering for patterns or information using techniques involving collaboration among various users, viewpoints, data sources, etc.

Examples include:

• Facebook (“your Friends liked…”)
• Amazon (“Users who viewed this product also viewed”, “frequently bought together”)
• LinkedIn (“People also viewed these profiles”)

Collaborative recommenders are better than content-based recommenders for recommending mixed content (imagine Pandora trying to recommend restaurants to a user based only on that user’s song preferences and listening history).

In practice, most modern recommender systems are a hybrid of the content-based and collaborative approach. and are a major source of revenue for online companies.

Example recommender system

The R package “recommenderlab” is a great toolset for creating your first recommender system. We will use the recommenderlab package for this example, although there are many different algorithms and packages for recommender systems. This package contains all the functions that you will need, plus the “Jester5K” sample dataset that we will be exploring.

From the documentation, we can see that “The data set contains a sample of 5000 users from the anonymous rating data from the Jester Online Joke Recommender System collected between April 1999 and May 2003.” Author disclaimer: these jokes can be very dated and do not reflect the views of the author nor Neal Analytics.

The following interactive recommender system was created by the author of the “recommenderlab” package, Michael Hahsler. First, you rate three jokes from -10 to 10. Then you can select the recommender algorithm that you would like to use, and the number of jokes that you want to be recommended to you.

The recommender will then recommend jokes and provide its predicted rating, i.e. the rating that it thinks you would score the joke. You can change out the set of jokes to rate and explore the recommender system.

The code underlying this recommender system can be view here.

How does a collaborative recommender system work?

A recommender system has a set of users and a set of items. A matrix is formed from the users and items so that every row is a single user and every column is a single item.

The goal is then to predict the ratings from a particular user for products that the user does not have ratings for, i.e. fill in the gaps to create a matrix of predicted interactions between each user and item.

Visually the structure of a classification dataset and a collaborative filtering dataset can be seen here:

Two forms of collaborative filtering are:

1. User-based
2. Item-based

User-based collaborative filtering (UBCF) relies on the assumption that users who have similar ratings will have similar preferences.

The first step in UBCF is to create “neighborhoods” of similar users. The mathematics behind creating a similarity matrix can get complicated but often rely on correlation. For more reading on in-depth mathematics, you can check out the original “recommenderlab” publication here.

Then, the average rating of the neighborhood for a particular item becomes the predicted rating for a user in that neighborhood without a rating for that item. The item with the highest predicted rating would then be recommended to that user.

 

Item-based collaborative filtering is very similar. Recommendations here are based on the similarity between items instead of users.

So, the assumption is that a user will prefer items that are similar to the items that they have already rated highly. Just as before, a similarity matrix is created between all the items.

The predicted rating for an item for a particular user is the weighted sum of the known ratings from that user for similar items.

Collaborative filtering in everyday life

A real-life human equivalent to this would be to ask your friend from the wine club, the ones you know usually like similar wines than you, to give you their thoughts on a new wine they’ve tested. If you ask five of these friends, and they all give a rating of 7 to 9, you know, with a fairly high degree of certainty that you will then probably also like it and give it a 7 to 9 rating.

Why recommender systems work

The answer lies in how each is used. Classification models, for example, will use independent variables to classify a dependent variable, e.g. age, gender, and class-number can be used to predict whether or not an individual survived the Titanic. The data is split into training and test sets. The model is trained on the training set and tested on the unseen test set. The goal is to predict a single interaction.

Now imagine a database with 1000 items and trying to classify whether a user will like item #1000 on a scale from 0 to 5 from whether or not they like items #1 through #999. The traditional classification/modeling process relies on the assumption that there is data on how the user feels about all the items. What happens if the user only has information regarding 2 items out of 1000?

This problem is called sparsity of the dataset. Sparse data is incredibly common due to the sheer amount of content that sites offer (think of the number of item listings on Amazon or the number of videos on YouTube).

When you have sparse data, a recommender system can help fill in the gaps. Have you ever noticed a website/platform asking you to rate items? It is trying to collect more information about your preferences as well as what your preference says about the item itself. Furthermore, when in doubt, a recommender system can recommend what is most popular in general as a baseline before it gets to know you.

Conclusion

There are many different algorithms for building a recommender system and various types of filtering. We have briefly touched on item-based collaborative filtering and user-based collaborative filtering, but there is much more out there.

The field is continually evolving and changing. Famously, Netflix offered a million-dollar prize to anyone who could create a better collaborative-filtering algorithm than their current system. To a business, optimizing their recommender can be synonymous with maximizing their revenue.

Learn more

Looking to learn more about how recommender systems can help your business? Check out Retail Product Recommendations, Neal Analytics whitepaper.

For more information on recommender systems, check out the publication behind the “recommender lab” package and Michael Hahsler’s other research publications. You can also read Recommender Systems from the Encyclopedia of Machine learning by Prem Melville and Vikas Sindhwani (IBM T. J. Watson Research Center).

This article was originally published 06/22/2020. You can also find it on LinkedIn.