Ever wonder how what you do online and around your house seems to be in full view of many who have no “need to know”? If so, it is likely that you are unfamiliar with the Algorithmically Driven Data Analysis at work in even the tiniest of devices in your home.
As Data Scientists, our work as mathematicians and programmers is principally focused on creating custom trading algorithms for proprietary use on trading desks in an Ultra-High Frequency, Low Latency environment where milliseconds mean the difference between profit and loss. The “flash crash” and Long-Term Capital incidents are examples of where programmed trading went horribly wrong, but most of the transactions that move Trillions of Dollars in Market Cap annually are completed properly.
Now this same technology is being applied to control and collect data flowing from the myriad of devices equipped with microchips organized into Embedded Systems. These tiny devices form the “Internet of Things” and the code used to command them uses virtually (no pun intended) the same math and programming languages that power the trading algorithms we spoke about earlier.
Although there is constant debate among Computer Scientists about whether Java is better than C++ for real-time processing, most of the chip level programs are written in C, especially for architectures like avr, msp430, xscale and some are programmed in assembler. Many applications such as Automatic Meter Reading use systems of hardware and software designed specifically for a particular device. WiFi makes it possible to host an environment where the data that comes off these devices can be analyzed and in some cases, commands be sent back to control temperature levels based on usage, time of day, or season.
Marketing Analysis has also adopted the use of algorithmic robots to gauge consumer sentiment over the web and suggest real-time strategies to take advantage of dislocations in buying patterns or patterns of buying behavior. One company providing these programmable robots for use across the social web is Kapow Software through their Kapplets™ technology.
Perhaps the most powerful and well-known use of complex algorithms is Google’s PageRank which works by assigning numerical weighting to every element within documents that has been hyperlinked. The purpose of the weighting is to quantify the relative significance of the elements in the set. Periodically, Google tweaks the algorithm and unleashes a wave of nasty surprises across the web as people find that their successful online niche businesses has suddenly – and unaccountably – disappeared.
While these “step-by-step procedures for calculations” seem unlikely candidates for the role of tyrant, their power comes from the fact that they are the key ingredients of all significant computer programs and the logic embedded in them determines what those programs do. In that sense, algorithms are the secret sauce of our increasingly digitized world.
A study by professors at Oxford University suggests that up to 47% of the jobs now performed by humans are at risk of being automated within the next twenty years. And if they haven’t already influenced your shopping, they have certainly determined how your pension fund is doing, and whether your application for a mortgage has been successful. One day they may even determine how you vote.