Are Recommender Systems Now Mainstream?

Overview

• Recommender systems are pioneered by some of the most successful and profitable technology focused businesses around the world.
• It is time for a wide range of businesses to adopt these concepts and embed them in their day to day business decision making models.
• Integration of recommender systems with enterprise applications have a huge potential of unlocking business value.
• The open source software movement has made enough progress to reduce the requirements of specialized skills to build and use such systems.

Context

Businesses like Facebook, Amazon, Netflix and Yahoo have mastered the art of building and evolving recommender systems. Needless to say that a significant part of their popularity, market valuation and more importantly profitability is positively impacted by these recommender systems.

Although these businesses have been using such systems for more than 20 years and have realized an enormous amount of value in terms of engaging their users/customers, making smarter business decisions, predicting future trends and capitalizing on those trends to increase market share; for some reason, businesses that do not consider themselves “Technology Companies” have failed to adopt these concepts and integrate them in their day to day business processes.

Argument 1 - We are not a technology company

This argument is presented time and again in various organizations, be it an insurance or re-insurance company, a pharmaceutical company or others.

Before anyone else, Amazon (who happens to have one of the best recommender systems in the world) as a retailer took full advantage of technology. Netflix, a company in the business of renting movies, completely changed their business model with the help of technology.

Indeed, Facebook and Yahoo are technology companies but BestBuy and Staples are not and neither is “Coursera”. All of these organizations have invested in technology (more specifically in effective recommender systems) and used these technologies to distinguish themselves from their competitors.

In today’s world, every company is a technology company. There was a time when technology was in place to support existing business processes. Times have changed. Today technology not only supports existing business processes but it enables businesses to identify new opportunities, create new business processes and in many cases, enables the creation of an entirely new business model.

Recommender system’s place in analytics eco-system

Typical questions posed are:

• Where exactly can such systems be used?
• Can they be part of internal interfaces as well or should they be designed only for external users?
• Should they be restricted only to certain types of users or can they be released to a wider user base
• internally?
• Should they be part of advanced analytics, data and decision science community, analyst community or
• can they be used by regular everyday business decision makers with limited technical knowledge?

A Recommender System is an enormously powerful tool that can not only help make right decisions but can also influence user behavior.  One of the most amazing characteristics of such a system is that it can be integrated with any other system, business process and transaction.

We all use services like Amazon.com to find out what others are saying about a specific item, product or service. Our decisions related to an item, whether favorable or not, are influenced by what other consumers are saying or what Amazon’s recommender system endorses, which is again based on what other users think and experience.

As human beings we always like to make a decision in collaboration with others. In my view, a recommender system is a powerful collaboration tool that can be used within organizations to make smart and meaningful business decisions.

Going back to the question, where should such systems be positioned with an organization’s analytics/information infrastructure portfolio?

It should be positioned within any information delivery channel that is used by people (users, customers & decision makers) to interact with the data. These channels can be any type of external or internal web interface, mobile interface, internal dashboard or scorecard, operational report or query and even business intelligence.

Examples

• A pharmaceutical firm designs a system that collects non-personalized “Tweets” about indications that various people are discussing. The system uses a “User-user collaborative filtering” technique to recommend investment & investigation opportunities in order to help people looking for help.
• An insurance company builds a system that looks at the internal profile of a new customer and compares it with other customers of similar type. The system looks at the history of those similar customers over the period of time and recommends the best possible policy and cost plan for this new customer.
• A sales director who is a business decision maker uses a business intelligence based ad-hoc query system to create an ad-hoc report based on certain filters and attributes. The embedded recommender engine in this ad-hoc system captures user actions, looks at the user’s profile and then compares it with other users of similar profiles (e.g. other sales director) to formulate a recommendation around other kinds of filters and attributes that can be added to the current query without revealing the data of other users.
• A pharmaceutical firm/CRO firm uses a recommender system embedded in their clinical trial management system. The clinical trial management system accepts and stores the data related to ongoing trials including data around the participant’s response on an ongoing basis. The embedded recommender system looks at this data and participant’s profile, matches it with similar participant profiles from the past(from successful and failed trials), identifies potential risks and recommends potential remediation steps based on steps taken by investigators in the past.

These are just a few examples of how a recommender system can be seamlessly integrated with other systems and can make that system smarter, more interactive and meaningful.

Argument 2 - Recommender systems are hard to build

Many would argue that such systems are hard to build and integrating them with other information delivery channels is a huge task that requires an enormous amount of specialized skills, time and effort.

On some level this argument is valid. Such systems require a very different kind of thought process, one that many mainstream businesses are not used to. Embedding these systems into other existing systems and then running analytics based on these systems would require a cohesive integration strategy.

The value that will come out of such systems however, can be a game changer. In terms of special skills required, I would argue that any organization that has already engaged in data science driven analytics already has the skills and change agents required to adopt such concepts.  Open source software movement has made it significantly simpler for organizations to experiment, evolve and adopt such systems.

Do we need to build systems from scratch?

Should you choose to experiment with recommender systems within the context of your own business model, a big question that you need to address is, “How do I start?”

You need a system that can accept a significant amount of data fairly quickly, store it, index it, build user profiles and finally integrate with user facing systems.  It will be really helpful if this system can also give you an option of implementing some standard algorithms out of the box.

You will also need a user “feedback loop” that can accept data from users and enhance pre-indexed data collected from other sources. This additional data can be used to make the system smarter.

One can certainly consider building a complete system from scratch but there are some other options out there as well.

One of my favorites is “Apache Solr”. “Solr” is an enterprise-ready system that can not only perform backend data engineering work but also implement several recommender system-relevant algorithms right out of the box.  It is easy enough to integrate with user facing systems (web based, desktop based and mobile based). It also integrates with several big data solutions such as various Hadoop Distributions, which makes it a perfect candidate to perform analytics.

For organizations interested in experimenting with recommender systems, “Solr” can be a great tool to get started.

Essential components of a recommender system

On a high level, a recommender system designed in the context of enterprise analytics and business intelligence should have the following components:

1. A robust data pipeline to collect and integrate data from various sources.
2. A scalable storage system that can store raw as well as indexed data.
3. An integration framework/approach to integrate these systems with other applications.
4. A passive feedback loop to understand user behavior (e.g. click streams being captured as the user interacts with the application).
5. Possibly, an active feedback loop (e.g. ask user preferences, ask users to complete their user profile, accept recommendation disagreement reasons) that can be used to make the system smarter.
6. An entity (item, product, dataset, KPI) to formulate the recommendation around.
7. Recommendation approach (e.g. User-User collaborative filtering/content based filtering).

Conclusion

There are many different ways a recommender system can be designed and used within enterprises. Choice of design, adoption of filtering approach and method of integration are things that need to be finalized in the context of a business situation; however, it is clear that recommender systems should be adopted more often than they are, within enterprises. By not doing so, the existing business decision- making framework, which is analytics and business intelligence-centric, is missing out on a big part of potential value proposition.