Businesses like Facebook, Amazon, Netflix and Yahoo have mastered the art of building and evolving recommender systems. Needless to say that a significant part of their popularity, market valuation and more importantly profitability is positively impacted by these recommender systems.
Although these businesses have been using such systems for more than 20 years and have realized an enormous amount of value in terms of engaging their users/customers, making smarter business decisions, predicting future trends and capitalizing on those trends to increase market share; for some reason, businesses that do not consider themselves “Technology Companies” have failed to adopt these concepts and integrate them in their day to day business processes.
This argument is presented time and again in various organizations, be it an insurance or re-insurance company, a pharmaceutical company or others.
Before anyone else, Amazon (who happens to have one of the best recommender systems in the world) as a retailer took full advantage of technology. Netflix, a company in the business of renting movies, completely changed their business model with the help of technology.
Indeed, Facebook and Yahoo are technology companies but BestBuy and Staples are not and neither is “Coursera”. All of these organizations have invested in technology (more specifically in effective recommender systems) and used these technologies to distinguish themselves from their competitors.
In today’s world, every company is a technology company. There was a time when technology was in place to support existing business processes. Times have changed. Today technology not only supports existing business processes but it enables businesses to identify new opportunities, create new business processes and in many cases, enables the creation of an entirely new business model.
Recommender system’s place in analytics eco-system
Typical questions posed are:
A Recommender System is an enormously powerful tool that can not only help make right decisions but can also influence user behavior. One of the most amazing characteristics of such a system is that it can be integrated with any other system, business process and transaction.
We all use services like Amazon.com to find out what others are saying about a specific item, product or service. Our decisions related to an item, whether favorable or not, are influenced by what other consumers are saying or what Amazon’s recommender system endorses, which is again based on what other users think and experience.
As human beings we always like to make a decision in collaboration with others. In my view, a recommender system is a powerful collaboration tool that can be used within organizations to make smart and meaningful business decisions.
Going back to the question, where should such systems be positioned with an organization’s analytics/information infrastructure portfolio?
It should be positioned within any information delivery channel that is used by people (users, customers & decision makers) to interact with the data. These channels can be any type of external or internal web interface, mobile interface, internal dashboard or scorecard, operational report or query and even business intelligence.
These are just a few examples of how a recommender system can be seamlessly integrated with other systems and can make that system smarter, more interactive and meaningful.
Many would argue that such systems are hard to build and integrating them with other information delivery channels is a huge task that requires an enormous amount of specialized skills, time and effort.
On some level this argument is valid. Such systems require a very different kind of thought process, one that many mainstream businesses are not used to. Embedding these systems into other existing systems and then running analytics based on these systems would require a cohesive integration strategy.
The value that will come out of such systems however, can be a game changer. In terms of special skills required, I would argue that any organization that has already engaged in data science driven analytics already has the skills and change agents required to adopt such concepts. Open source software movement has made it significantly simpler for organizations to experiment, evolve and adopt such systems.
Do we need to build systems from scratch?
Should you choose to experiment with recommender systems within the context of your own business model, a big question that you need to address is, “How do I start?”
You need a system that can accept a significant amount of data fairly quickly, store it, index it, build user profiles and finally integrate with user facing systems. It will be really helpful if this system can also give you an option of implementing some standard algorithms out of the box.
You will also need a user “feedback loop” that can accept data from users and enhance pre-indexed data collected from other sources. This additional data can be used to make the system smarter.
One can certainly consider building a complete system from scratch but there are some other options out there as well.
One of my favorites is “Apache Solr”. “Solr” is an enterprise-ready system that can not only perform backend data engineering work but also implement several recommender system-relevant algorithms right out of the box. It is easy enough to integrate with user facing systems (web based, desktop based and mobile based). It also integrates with several big data solutions such as various Hadoop Distributions, which makes it a perfect candidate to perform analytics.
For organizations interested in experimenting with recommender systems, “Solr” can be a great tool to get started.
Essential components of a recommender system
On a high level, a recommender system designed in the context of enterprise analytics and business intelligence should have the following components:
There are many different ways a recommender system can be designed and used within enterprises. Choice of design, adoption of filtering approach and method of integration are things that need to be finalized in the context of a business situation; however, it is clear that recommender systems should be adopted more often than they are, within enterprises. By not doing so, the existing business decision- making framework, which is analytics and business intelligence-centric, is missing out on a big part of potential value proposition.