Historically, most of the analytics and decisions support systems processed data in a high-latency, batch-oriented environment.
This is not only because of limitations around computing abilities at the time, such as storage, memory, network bandwidth and lack of wireless communications enabled end-points, but also because of the way global economy functioned in the past.
Things were less connected and less dependent on each other in the past. Various kinds of data sources generated data that was limited, predictable and manageable.
Demand of business was to make decisions in an accepted timeframe which could span from one week to three months.
The last 10 years saw old world crumbling and last three years saw that world disappearing quickly.
Global economy is a lot more interconnected then it was before, speed of innovation has challenged and overturned every possible known principal around making a business decisions based on historical data.
Cost pressures, increasing competition and use cyber-attacks as a weapon did not help the cause either.
On the other hand, computing abilities have increased significantly in the last 15 years. Distributed or cluster computing is accessible to everyone now and microprocessors have become so small yet powerful that they can be put inside a device smaller than your thumb.
With all of these changes, concept of real-time systems, streaming systems, making decisions in real time and removing or reducing the human involvement in certain decision making processes and allowing machines to talk to each other and make decisions by themselves became extremely profound.
Systems that were capable of communicating with each other in real time have been around for many years. People who made critical decisions on a stream of real-time data have been around for long time, as well.
These devices were extremely expensive to implement and were extremely limited in terms of their ability, making them accessible only to government and a handful of major corporations.
All that changed in last 15 years.
There is no one example that can be identified which acted as a catalyst for wider adoption of real-time systems.
If I have to pick one, I would use Netflix as one of the many examples. A business that started many years ago with the vision that not all people may like to go to a store to rent movies, a lot of them might appreciate movies coming to them.
Netflix successfully tested this philosophy and challenged the in-store renting business model for movie-renting businesses. As a result, they invented a new business model.
Sometime soon after that, they realized that technologies are available to reduce their customer’s wait time from two or three days to few minutes. They challenged their own business model— in a way they ridiculed it—and invented a new model that was completely based on streaming movies directly to customer’s devices. A profound example of sending data to devices in real time, capturing user interaction in real time and making decisions based on these interactions in real time.
As real-time processing design patterns moved from hype to substance, many use cases have emerged in various organizations operating in several verticals. Some of them are:
We are still in early stage of commoditizing real-time commuting systems, but it is not too hard to realize that these systems are already making a huge impact.
From fraud detection to understanding user behaviors to listening to social media and improving products and services to reducing the cost and turnaround time of clinical trials, the list is long.
In coming years, we will see more and more systems operating in real time. We will also see more machines talking to each other, exchanging data, making decisions and taking actions.
In my view, real-time computing, analytics and stream processing have bigger positive impacts than any other design pattern, concept or process that came out of big data and analytics technologies.
These systems and associated technologies will get only better with time. They eventually will play an enormously important role in realizing IoT vision for all of us in the near future.
In Part 2 of this article series, I will discuss various architectural patterns, system design principals and various technologies that are available to build such systems.