An Analytics Journey – Big or Small – Needs the Right Data

In order to succeed in a digital world, companies must get out of inertia and negative complexity. Instead, companies must focus on the future from a digital perspective and working backwards. This includes changing retrograde accounts for predictive analytics, combined with experimental data-driven strategy.

Yet even then, the data may not contain the proper answer. The combination of some data and aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data, no matter how big the data are. Moreover, in several examples, data correlation is not causation.

In the more traditional, mainstream companies — firms that have built and cultivated their management teams by heavily valuing intuition and experience as a way to make decisions — the executives don’t understand the statistical black box that comes with the advanced analysis. No one is advocating using analytics as a replacement for judgment and intuition. Rather, analytics has the potential to become a much more powerful aid to judgment and intuition.

Data management readiness

Big data is commonly defined based on 5 Vs:

1. Large Volumes of data
2. Variety of data formats
3. High Velocity of processing
4. Veracity,  referring to the messiness or trustworthiness of the data
5. Value, referring to our ability turn our data into value

Big data and analytics technology now allows us to work with these types of data. The volumes often make up for the lack of quality or accuracy. The common notion is that bigger is often better when it comes to data and analytics, which implies a perception of high complexity and high investment required. But this is not always the case.

Michael Stonebraker, a renowned database researcher, Turing Award laureate and adjunct professor at MIT's Computer Science and Artificial Intelligence Laboratory, presented in his research that 69% of corporate executives named greater data variety as the most important factor. Therefore, many firms see the big opportunity in Big data resulting from the capture of traditional legacy data sources that have gone untapped in the past. These are data sets that have typically sat outside the purview of traditional data marts or warehouses — the “long tail” data.

When performing a large-scale data governance implementation, there are many critical preparations that need to be completed first — most importantly, you’ll need to establish data quality enough for business purpose. Without clean, quality data in the system, the real-time capabilities available through any technologies will not deliver the correct results. Rather, it will simply end up delivering the wrong information faster.

Possibly the most difficult task in any large-scale data migration project is establishing when the data is business-ready. This means that the data is not only ready-to-load, but that it will also generate the desired result in the new target real-time system. Typical processes to determine data readiness involve creating and running scripts and execution logic along with analyzing error reports for failures and required changes.

• Analytics and moreover, advanced analytics are based on data and algorithms.
• Decision analytics supports human decisions with visual analytics the user models to reflect reasoning.
• Descriptive analytics gains insight from historical data with reporting.
• Predictive analytics employs predictive modeling using statistical and machine learning techniques.
• Prescriptive analytics recommends decisions using optimization and simulation. In general, there is poor competence within the organizations using predictive analytics, which algorithms are much more complex than observed in a Business Intelligent models based on semantic layer and structured data (descriptive analytics).

Data science capabilities

The first step is not looking for Big data solutions to follow the crowd. And also not looking to be so impressive with the big and complex cases presented as most of have taken a long journey before obtaining achievements. It is necessary to recognize that the company should be managed based on data, which implies some culture and profound mindset shifts. In addition, there are serious legal and regulatory risks around issues such as data privacy that can result in breach of trust. Most consumers do not want companies to share personal information, and many say that they are less willing to share personal information in the future.

In general, you should start looking for smaller, quicker wins, but make sure you get the right data that addresses the strategic problem you're trying to measure or compare, not just the data that's easiest to obtain. While this can speed up a project, the analytic results are likely to have only limited value, which can jeopardize the whole program.

The basic ways that data can be obtained are from APIs, from databases and from colleagues in various formats. Tidy data dramatically speeds downstream data analysis tasks. The components of a complete data set include raw data, processing instructions, codebooks and processed data supporting collecting, cleaning and sharing data. You should ask the right questions to manage data sets, making inferences and creating visualizations to communicate results.

Traditionally, data visualization or reporting is developed based on a waterfall approach where typically one expert provides the requirements in the earlier phases of the process and verifies the final result near the end. In big data, the agile approach is more adequate as multiple experts and IT teams must work together toward the solution in a timely manner with defined iterative job tasks, requiring much more engagement and availability to the project.

Applying analytical insights

The extensive usage of key process indicators and dashboards have escalated the adoption of BI platforms across industries. However, the transition from big data pilots to organization-wide deployments can be difficult given costs, effort required and broader questions about whether the analytics will be used consistently by decision makers.

Most companies noted several difficulties applying analytical insights — not using analytics to drive strategic decisions, uncertainty about how to apply analytics and failure to act on insights. Over the years, access to useful data has continued to increase. As the volume and complexity of data grows at exponential rates, companies wrestle with how to turn the data into useful insights that can guide the business.

While technology is a critical aspect, developing strong data science capabilities within the organization — deep knowledge in statistics, software development programming and of the specific domain of the problem to be solved — is equally important. A data scientist is in high demand nowadays, and it is not easy to hire and retain high-caliber talent. Ultimately, expanding data science capabilities should closely follow the executives’ change of mindset in how to make decisions based on data and algorithms.