What is Dark Data in Big Data?

Previously we talked about Big Data through various articles. But, What is Dark Data? Dark Data is Big Data Which is Collected, Processed & Stored For Need But Organization Fails to Use it For Direct of Indirect Monetizing.

What is Dark Data in Big Data in Plain English?

Dark data is also known as dusty data and opposite side of the spectrum of light data. It is Gartner who first coined the terminology as Dark Data. Dark data is a set of type unstructured, untagged data that which can be found in the data repositories and has not been analyzed. It is similar to big data but not used by the IT administrators due to lesser return of investment. Source of dark data can be information gathered by sensors, server log files, data archives and other telematics, past customer information, previous employee information, raw survey data, financial statements including data from ATM, email correspondences, old versions of documents.

An organization may be aware that the dark data is being collected. About 90% of data generated by the sensors never gets utilized. Dark data may be important to analyze to aid business decision making. Typically dark data is either complex to analyze or is stored in locations where analysis is difficult, like server log files can have spam, brute force IP address which can save time to fight DDoS but it is impractical to collect and analyze such data from all the servers on this earth. The overall process can be costly.

What is Dark Data and How it is Important?

Dark data often has legal importance. That is called lopsided priorities. Take the example of a bank analyzing ATM transactions of past 7 years. ATM and debit card team is thoughtful about customer details but not to the data on who was in charge of security of the ATM. That is unattended data and could have provided valuable insights but there is no priority assigned to that aspect as goals are set for a work.

Dark data can cause legal, financial and other troubles out of data leak, data theft. Companies are already facing issues of keeping dark data. Dark data has importance in big data analytics and research which can lead to another industrial revolution.