We are in the midst of a Big Data vortex. The information from Big Data is impacting the ways we live and work, and it is said to be changing the way we think. Big Data quickly becomes the center of every business. Massive volumes of data arriving from multiple sources make people sit up and pay attention.
While Big Data is on the rise, and we trust it can help us reveal hidden truths about our world, there remain many questions related to Big Data, such as how to store, analyze and use such large data sets.
Let’s set aside the technical questions for now and think of one big question – when the world is brimming with data, and we can achieve accurate and timely decision making in a revolutionary level that was not possible to achieve before, will human judgment still be necessary?
This is one answer from Andrew McAfee, the co-director of the Initiative on the Digital Economy in the MIT Sloan School of Management. He introduced the theory above as “the rule for the second machine age.” McAfee also believed that as data becomes more available everywhere, computers become more powerful and predictive algorithms become more advanced, we will see the end of decisions made by human judgment.
At the same time, the arguments of his opponents are also persuasive.
As Thomas H. Davenport said above, Big Data cannot totally replace human judgment. However, they can work together and complement each other.
It is important to note that “Big Data” is just a fancy word; it is still data. It follows the same rules as “small data” to create information – filter out bad data, conduct statistical analysis, run the right analytics, create nice visualization and produce relevant insights.
Here are several considerations to examine before answering this big question:
Big Data is big in volume, but it is not always accurate. Most Big Data being collected today from our social interactions is from purely observational sources. Since these sources do not strictly follow statistical design rules, the resulting Big Data might lead to “precisely inaccurate” results – the collected data values are close to each other, but far from the true value. Before using any data sets, analysts need to evaluate their accuracy. This is human judgment.
When you have a small set of data, it is easy to explain exactly what it consists of. When the data becomes bigger, it becomes much more difficult to tell what it represents. Unless we have the entire population and no sampling consideration is required, the foundation of data analysis still relies on statistics. To generate meaningful information to support a decision, analysts need choose a reasonable sample and good analytical methods to mitigate bias. This is human judgment.
This is a very thought-provoking article title from author Bernard Marr. He uses one simple sentence to summarize this situation. Let’s assume Big Data can answer everything. But what do you want or need to know? To get the right answers, decision makers need to work with analysts to define the problems and design great questions to ask the data. This is human judgment.
Big Data doesn’t generate magic answers automatically; it relies on analysts to choose the right methods and decide whether or not to trust the machine-generated results. The massive data could become messy or magic, depending on the way one handles it. If one’s common sense discerns that the results are incorrect, there is a good chance they are. This is human judgment.
The key benefit of Big Data is not about data, it’s about information. “Data becoming available everywhere” does not have the same meaning of “Useful information becoming available everywhere.” Where information is not available or does not provide clear direction, we need to use our knowledge and experience to bridge the gap. This is human judgment.
Powerful computers and sophisticated analytical tools are able to automate data analysis and build a correlation between attributes, but they are not able to automate common sense and human judgment. In many cases, Big Data is just providing better information and helping direct us to make good decisions. This is human judgment.
Whether data sets are big or small, we need to secure sensitive data and protect private information. Organizations need to create comprehensive strategies to use information from Big Data while avoiding the pitfalls. When data becomes available everywhere, the strategy will be focused on, “What data can be used?” rather than, “How can we find data?” This is human judgment.
The real art of decision-making is to achieve optimum balance between evidence and judgment. The key here is to use evidence from Big Data to support decisions (to justify whether a decision is good) or inform decisions (a combination of evidence, intuition and experiences), rather than to make the decisions. With the increasing availability of Big Data, we need to be able to balance risks and opportunities on whether to use evidence over judgment to make decisions. And once again, you guessed it – this is human judgment.
Article written by Jason Li
Want more? For Job Seekers | For Employers | For Contributors
