We are witness to and part of an Internet revolution that is tightening its grip over the neck, mind and entire body of an individual, society and enterprise. The level of change has not been experienced in the last five decades or maybe last century! Enterprises need better insight and visibility about their customers. They must understand how customers are connected to other customers, places and things – i.e. ‘net worth of their network’
An effective data management strategy helps an enterprise with a multi-dimensional view of their customers and also helps improve their risk management and operational metrics. The biggest enabler to this strategy is enterprises realizing the importance of data assets that they hold.
The sooner we start realizing the importance of the biggest asset we hold, it will be relatively easier to embrace the change and keep peace with the changing pace.
What is Data?
Data is the most valuable asset to an enterprise. It helps an organization to learn from the past, act in the present and prepare for the future.
93 percent of companies that lost their data center for 10 days or more due to a disaster filed for bankruptcy within one year of the disaster. 50 percent of businesses that found themselves without data management for this same time period filed for bankruptcy immediately.
– National Archives and Records Administration in Washington, D.C.
In our routine life, we use the words data and information interchangeably, but this is not correct. Data is a collection of facts and observations. Data becomes information when these are processed. The processing of data includes:
- Cleaning data from errors and reducing sources of unreliability
- Analyzing data to make it relevant to decisions at hand
- Organizing data in ways that help understanding
Looking back only a year or two ago, data related mostly to Microsoft Word documents, Excel spreadsheets, PowerPoint files, relational structures stored in a RDBMS database, etc. Thanks to the Internet revolution and smartphones, there is a huge spurt in data growth. 90 percent of data in the world is generated in the last two years. So, apart from humans providing the traditional sources of data (i.e. from business mobile apps), there are massive contributions in the form of blogs, reviews, messages, images and machines generating data as a byproduct of interacting with people and with other devices and machines.
Figure 1 – Imperatives Surrounding Data
With such data growth and volumes, naturally it will be expected for a machine to treat a video or audio file at par to a Word document. There will not be an extra step of translation or transformation required to bring a video file in a compatible format for reading. In other words, we require a data processing framework that can make a machine intelligent or comparable to humans.
“When machines gets intelligent, we may not even notice we will be them
and they will be us.”
– Big Data Strata Conference
6 Key Considerations for a Next Gen Data Architecture
1. Data Quality
“80 percent of work in any data project is in cleaning the data.”
– DJ Patil, Chief Data Scientist of the U.S. Office of Science and Technology Policy
Data quality is proportional directly to trust and reliability. Considering data volumes and varied data sources (internal, external), ensuring data quality is of utmost importance. Goals to achieve enterprise data quality are:
- Make data available consistently at all times
- Reduce cost of rework
- Complete availability of data at all times
- Clearly define rules that can measure the accuracy of data
2. Data Governance and Security
- Define rules to pull and push the data in a data center, data lake or application
- Document ownership and responsibility toward data
3. Business and Tech Working Toward a Common Objective
- Require a Business Data Analyst position in an enterprise to take care of a common objective
- Recognize huge value locked in internal data
- Consider adopting new technologies to unlock the value
4. Effective Data Management Strategy
- A strategy is important in the age of growth, mergers, acquisitions and realignment
- It gives a complete 360 degree view of a customer
- It improves operational performance and excellence
5. Omnipresence of Data
- Be aware of the numerous sources of data
- Incorporate and nurture a culture to accept new data
- Plan a 40 percent headroom w.r.t system capacity, storage
- Remember data multiplication rate is far and exponentially higher than processing capability
- Constantly re-visit the rules to carve out relevant data and separate noise
- Strike a balance between opportunity versus challenge
- See data ‘value’ as the key
- Understand that data challenges vary by industry
Source: Bain & Company
6. Keeping Pace with Technology
- Considering today's tech pace, be prepared for an average data architecture to hold good for two to four years and then it will require a change
- Design an architecture that can accommodate the evolution of technology
- Make it elastic and scalable
- Use a flexible data model
- Manage real-time and batch scenarios simultaneously as needed
Source: In-Memory Computing by Prof. Dr. H.C. Hasso Plattner
