A recommender system, also called a recommendation engine or content discovery platform is a type of information filtering system that aims to suggest items most relevant for some input or to a particular user. In the context of social media, search engines, and other online services, a recommender system for a given service might sometimes informally but erroneously be referred to as the service's "algorithm". The use of recommender systems is pervasive, with commonly recognised examples including the generation of playlists for video and music streaming services, product recommendations for e-commerce platforms, and topics and individual items posted on social media platforms and the open web. Online services employ machine learning or specialized deep learning recommendation models (DLRMs) that analyze user behavior and preferences to generate personalized content feeds for millions of users at any moment. The value of these systems becomes particularly evident in scenarios where users must select from a large number of options, such as products, media, or content. Suggestions are typically designed to shorten or work as a substitute for decision-making processes, including the selection of a product, musical selection, or online news source to read.
Overview Recommender systems usually make use of either or both collaborative filtering and content-based filtering, as well as other systems such as knowledge-based systems. Collaborative filtering approaches build a model from a user's past behavior (e.g., items previously purchased or selected and/or numerical ratings given to those items) as well as similar decisions made by other users. This model is then used to predict items (or ratings for items) that the user may have an interest in. Content-based filtering approaches utilize a series of discrete, pre-tagged characteristics of an item in order to recommend additional items with similar properties.
Example The differences between collaborative and content-based filtering can be demonstrated by comparing two early music recommender systems, Last.fm and Pandora Radio.
Last.fm creates a "station" of recommended songs by observing what bands and individual tracks the user has listened to on a regular basis and comparing those against the listening behavior of other users. Last.fm will play tracks that do not appear in the user's library, but are often played by other users with similar interests. As this approach leverages the behavior of users, it is an example of a collaborative filtering technique. Pandora uses the properties of a song or artist (a subset of the 450 attributes provided by the Music Genome Project) to seed a "station" that plays music with similar properties. User feedback is used to refine the station's results, deemphasizing certain attributes when a user "dislikes" a particular song and emphasizing other attributes when a user "likes" a song. This is an example of a content-based approach.
Each type of system has its strengths and weaknesses. In the above example, Last.fm requires a large amount of information about a user to make accurate recommendations. This is an example of the cold start problem, and is common in collaborative filtering systems. Whereas Pandora needs very little information to start, it is far more limited in scope (for example, it can only make recommendations that are similar to the original seed).
Alternative implementations Recommender systems are a useful alternative to search algorithms since they help users discover items they might not have found otherwise. Of note, recommender systems are often implemented using search engines indexing non-traditional data. In some cases, like in the Gonzalez v. Google Supreme Court case, may argue that search and recommendation algorithms are different technologies.
Recommender systems have been the focus of several granted patents,Stack, Charles. "System and method for providing recommendation of goods and services based on recorded purchasing history." U.S. Patent 7,222,085, issued May 22, 2007.Herz, Frederick SM. "Customized electronic newspapers and advertisements." U.S. Patent 7,483,871, issued January 27, 2009.Herz, Frederick, Lyle Ungar, Jian Zhang, and David Wachob. "System and method for providing access to data using customer profiles." U.S. Patent 8,056,100, issued November 8, 2011.Harbick, Andrew V., Ryan J. Snodgrass, and Joel R. Spiegel. "Playlist-based detection of similar digital works and work creators." U.S. Patent 8,468,046, issued June 18, 2013.Linden, Gregory D., Brent Russell Smith, and Nida K. Zada. "Automated detection and exposure of behavior-based relationships between browsable items." U.S. Patent 9,070,156, issued June 30, 2015. and there are more than 50 software libraries that support the development of recommender systems including LensKit, RecBole, ReChorus and RecPack.
History
Elaine Rich created the first recommender system in 1979, called Grundy.BEEL, Joeran, et al. Paper recommender systems: a literature survey. International Journal on Digital Libraries, 2016, 17. Jg., Nr. 4, S. 305–338.RICH, Elaine. User modeling via stereotypes. Cognitive science, 1979, 3. Jg., Nr. 4, S. 329–354. She looked for a way to recommend users books they might like. Her idea was to create a system that asks users specific questions and classifies them into classes of preferences, or "stereotypes", depending on their answers. Depending on users' stereotype membership, they would then get recommendations for books they might like.
Another early recommender system, called a "digital bookshelf", was described in a 1990 technical report by Jussi Karlgren at Columbia University, Karlgren, Jussi. "An Algebra for Recommendations.. Syslab Working Paper 179 (1990). " and implemented at scale and worked through in technical reports and publications from 1994 onwards by Jussi Karlgren, then at SICS,Karlgren, Jussi. "Newsgroup Clustering Based On User Behavior-A Recommendation Algebra ." SICS Research Report (1994). and research groups led by Pattie Maes at MIT,Shardanand, Upendra, and Pattie Maes. "Social information filtering: algorithms for automating "word of mouth"." In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 210–217. ACM Press/Addison-Wesley Publishing Co., 1995. Will Hill at Bellcore,Hill, Will, Larry Stead, Mark Rosenstein, and George Furnas. "Recommending and evaluating choices in a virtual community of use ." In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 194–201. ACM Press/Addison-Wesley Publishing Co., 1995. and Paul Resnick, also at MIT,Resnick, Paul, Neophytos Iacovou, Mitesh Suchak, Peter Bergström, and John Riedl. "GroupLens: an open architecture for collaborative filtering of netnews." In Proceedings of the 1994 ACM conference on Computer supported cooperative work, pp. 175–186. ACM, 1994.Resnick, Paul, and Hal R. Varian. "Recommender systems." Communications of the ACM 40, no. 3 (1997): 56–58. whose work with GroupLens was awarded the 2010 ACM Software Systems Award.
Montaner provided the first overview of recommender systems from an intelligent agent perspective.. Gediminas Adomavicius provided a new, alternate overview of recommender systems.. Herlocker provides an additional overview of evaluation techniques for recommender systems,. and Beel et al. discussed the problems of offline evaluations. Beel et al. have also provided literature surveys on available research paper recommender systems and existing challenges.
Approaches
Collaborative filtering
One approach to the design of recommender systems that has wide use is collaborative filtering. Collaborative filtering is based on the assumption that people who agreed in the past will agree in the future, and that they will like similar kinds of items as they liked in the past. The system generates recommendations using only information about rating profiles for different users or items. By locating peer users/items with a rating history similar to the current user or item, they generate recommendations using this neighborhood.
The collaborative filtering approach does not rely on machine analyzable content and therefore it is capable of accurately recommending complex items such as movies without requiring an "understanding" of the item itself. Many algorithms have been used in measuring user similarity or item similarity in recommender systems. For example, the k-nearest neighbor (k-NN) approach, and the Pearson Correlation as first implemented by Allen.
Collaborative filtering approaches often suffer from three problems: cold start, scalability, and sparsity.Sanghack Lee and Jihoon Yang and Sung-Yong Park, Discovery of Hidden Similarity on Collaborative Filtering to Overcome Sparsity Problem, Discovery Science, 2007. Cold start: For a new user or item, there is not enough data to make accurate recommendations. Note: one commonly implemented solution to this problem is the multi-armed bandit algorithm. Scalability: There are millions of users and products in many of the environments in which these systems make recommendations. Thus, a large amount of computation power is often necessary to calculate recommendations. Sparsity: The number of items sold on major e-commerce sites is extremely large. The most active users will only have rated a small subset of the overall database. Thus, even the most popular items have very few ratings.
Social networking services have used collaborative filtering to recommend new friends, groups, and other social connections by examining the network of connections between a user and their friends.
Content-based filtering
Another common approach when designing recommender systems is content-based filtering. Content-based filtering methods are based on a description of the item and a profile of the user's preferences. These methods are best suited to situations where there is known data on an item (name, location, description, etc.), but not on the user. Content-based recommenders treat recommendation as a user-specific classification problem and learn a classifier for the user's likes and dislikes based on an item's features.
In this system, keywords or, for images and video, vision embeddings for representation of the content as vectors similarly to vision language models or other deep learning-based computer vision techniques, are used to describe the items, and a user profile is built to indicate the type of item this user likes according to the embeddings the user spends the most time on. In other words, these algorithms try to recommend items similar to those that a user liked in the past, spends the most time on, or is examining in the present. It does not rely on a user sign-in mechanism to generate this often temporary profile. In particular, various candidate items are compared with items previously rated by the user by likes or watch time, and the best-matching items are recommended. This approach has its roots in information retrieval and information filtering research. Deep learning-based content filtering systems are the most pervasive and effective recommendation systems, used by major social media platforms such as Meta's and X/Twitter. They do not require users to manually tag posts with hashtags. Posts can be associated with users by comparing their embeddings with content embeddings via cosine difference. In its present form, it uses a specialized Deep Learning Recommendation Model (DLRM) that uses several billion parameters, up to 150 Billion for Meta's which is similar in size to some contemporary Large Language Models. X/Twitter's recommendation model uses a transformer, not unlike contemporary Large Language Models. These are run on contemporary general purpose graphical processing units, or custom processors.
To create a user profile, the system mostly focuses on two types of information:
A model of the user's preference.