Article Preview
Top1. Introduction
Data being the raw material for information is a collection of symbols, text, image, video or any other form. It is further processed by applying certain operations or set of instructions for removing all types of ambiguity to retrieve concrete information. Data can be of any type like structured, unstructured and semi-structured that can be generated by both humans and machines. Structured data is one that can be stored, accessed and processed in a well-defined structured format. A student or an employee table in a database is an example of structured data. Unstructured data is present in an unknown form and has an unstructured classification - for example, a combination of images, text files, audio, videos, etc. And the last one is a semi-structured type of data. This form can be in a structured and unstructured format as well. Data represented in an XML file, JSON documents, and NoSQL databases are examples of a semi-structured state.
Big data is a concept or form of data having a very large volume, which keeps growing exponentially with the time. It deals with the enormous and complex data sets which cannot be processed or managed with the conventional data processing software or any other tools. Big data analytics has the potential to transform and provide insight to every business or field. Earlier, data was stored in Megabytes or Gigabytes, but today, a large amount of data is producing ceaselessly in terms of Petabytes (PB) or Zettabytes (ZB) which eventually requires huge storage area and management.
According to some facts about data generation, videos of 300 hours in length are uploaded to YouTube in every minute and views performed every single day are almost 5 billion (Sagiroglu and Sinanc 2013). Each day, 400 hundred million tweets are sent (Jannapureddy et al. 2019). On average, people post about 500 million tweets per day (Sagiroglu and Sinanc 2013). Data processed by Google is in terms of petabytes (Chen, Mao, and Liu 2014). Over 10 PB of log data is generated by Facebook every month (Chen, Mao, and Liu 2014). Within the next decade, an increase in the amount of information will be around 50 times of current number and the information technology specialists’ number will go up by 1.5 times (Sagiroglu and Sinanc 2013). Ninety percent of the whole data currently available is the amount of big data created from its different sources in the last two years (Jannapureddy et al. 2019). Social media, banks, business processes, web servers, instruments, websites, stock markets, emails, health records, medical image data (Chatterjee et al. 2018), search queries, logs, sensors, scientific data, online transactions, videos, audios, images, financial services, retail, text document, photography, mobile phones are some of big data sources in this electronic era. Big data trend is leading in many areas such as improving healthcare and public health, understanding and targeting customers, understanding and optimizing business processes and many other fields like improving science and research, optimizing machine and device performance, improving security and law enforcement.
The significance of big data does not spin around the size of the data. Retrieval of useful and accurate information by processing and analysing the data for better outcomes, smart decision making, early detection of errors, cost, and time reductions make it significant. To manage it precisely, we need some other platform. One of the optimal solutions to this big data problem, we will be addressing in this paper is the ‘Hadoop’ cluster and Hadoop Distributed File System (HDFS) as its storage platform on ‘cloud’ platform to manipulate the data. Hadoop can be used to manage the big data using cloud servers, but without Hadoop, cloud alone cannot handle big data.