The year was 1961. Computers were still in their infancy, and the race to the moon was just beginning. Edward Lorenz, an MIT meteorologist, was developing a weather-prediction model. Lorenz theorized that a minuscule occurrence, such as a tiny butterfly flapping its wings in the Amazon, could hypothetically set in motion a chain of events that could cause tornadoes to touch down in Texas a few days later.
Fast forward to present day. Operational big data applications have predictive analytics at their core, and the business may be riding on a predictive analytics infrastructure. So, how stable is this infrastructure under the chaotic, complex, dynamic conditions that you may find in many operational environments? How do we effectively take care of the risks?
Predictive models are usually built around specific scenarios with well-defined dependent and independent variables in expected distributions. Consider the challenge of monitoring any new application brought into the environment being similar to predicting the behavior; there is a sensitive dependence on initial conditions based on their own predispositions, which creates interdependencies on future actions that can be difficult to vaticinate.
Predictive models thrive on linear relationships among variables, but what happens when the underlying reality being modeled becomes non-linear – in other words, when seemingly inconsequential new events, neither expected nor modeled explicitly, render formerly powerful predictive models impotent? How does one take predictive actions against the risks?
There are many mathematical techniques for modeling non-linear relationships, but given that these are highly specialized and often unfamiliar to business-oriented data scientists, you probably haven't considered them into the sorts of predictive models.
The next frontier on operational big data applications is the Internet of Things (IoT), which is likely to become a deepening vortex of butterfly effects waiting to happen. This is due to the fact that non-linear effects are likely to be far more prevalent in IoT environments – such as smart grid and real-time distributed process monitoring.
Please give yourself a second to think about the arising security risks with the Internet of Things.
The chief causes for all these questions and effects will be the continual expansion in new IoT endpoints and growth in these endpoints' generation and consumption of a wider range of messages under a broader range of operational scenarios. If nothing else, the sheer combinatorial explosion in IoT interaction patterns is a recipe for chaotic traffic loads and unpredictable security breach.
Think about it. Every new sensor, gadget, system and other device that enters the IoT becomes yet another butterfly, and every new piece of data it emits or action it takes becomes another flapping of the butterfly's wings. Throughout the world, as more of these butterflies come online, produce and consume more data in countless combinations in every possible circumstance, the non-linear effects are almost certain to intensify.
The vision of a planet-wide optimization depends on keeping the butterfly effect under control.