Big Data: An Emerging Field of Data Engineering

Introduction

Big data is concept more uses from various companies. Some examples are related with oil and gas refineries and mining industries, online social networks, multimedia data and business related transactions. More amount of data collected from different increasingly efficient data storing various devices as well as stored on fast-growing mass storage, people are keep to search to find solutions to collect and process the information more efficiently, and to find various values from the mass at the same time. When referring to big data research strategy problems, people often support the 4 v's -- volume, velocity, variety, and value. These pose support more brand-new challenges to computer scientists nowadays (Ahmed & Karypis, 2012).

In 2004, Wal-Mart claimed to have the very large data warehouse with approx 500 terabytes storage. In 2009, eBay announce storage amounted to eight PB. Two years later, the Yahoo warehouse totaled 170 PB. Since the rise of digitization, various technical enterprises have amassed burgeoning amounts of digital information, including trillions of bytes of data about their customers, suppliers and operations (Aral & Walker, 2012). Information volume is also growing exponentially due to the spread of machine-oriented data (data records, web-log files, sensors information) and from growing human involvement within the social networks.

Basically the growth of information will never stop. As per the 2011 IDC Digital Universe Study, 130 EB of data were created and stored in 2006. The amount grew to 1,327 EB in 2011 and is projected to grow at 43.6% to 7,913 EB in 2016. The growth of information constitutes the “Big Data” concept – a technological concept bring about by the rapid rate of data expands and parallel generalization in technology that have given rise to an contribution of software and hardware products that are enabling users to analyze this information to produce new and more granular desire levels of information (Machanavajjhala & Reiter, 2012). A decade of Digital Universe Growth: Storage in Exabytes has been illustrated in figure 1.

Figure 1.

A decade of Digital Universe Growth: Storage in Exabytes (Machanavajjhala & Reiter, 2012)

According to McKinsey, Big Data, perform to datasets whose sizes are beyond the ability of database software product to acquire, store, manage and study. There is no explicit definition of how big an image dataset should be in order to be considered large Big Data. New technology has to be manages this Big Data concept. Basically, IDC defines Big Data technologies considered as a new generation of database technologies and architectures designed to extract data value from large volumes of a wide variety of data. According to Reilly, “Big data is information which is exceeds the processing capacity of traditional database products. Basically, information is too big, moves too fast, or does not assemble the structures of desired database architectures. To gain value from these databases resource, there must be different way to process it” (Machanavajjhala & Reiter, 2012).

Characteristics Of Big Data

Big Data is not just about the volume of information but also includes information variety and velocity. Together, these three parameters form the three Vs of Big Data theory. Volume is synonymous form of the “big” in the term, “Big Data”. Volume is a relative term – some smaller-sized industries are likely to have mere gigabytes or terabytes of information storage as opposed to the PB or EB of data that big data global enterprises have. Data volume will continue to expand, regardless of the organization’s capacity. There is a natural definition for companies to store data of all categories: account data, biomedical data, research data and so on. Many of these organization datasets are within the terabytes range today but in early days they could be reach PB or even EB (Aral & Walker, 2012). Data can come from a variety of resources and in a variety of category. With the expansion of sensors, various smart devices such smart phone as well as social networking sites, information in an organization has become more complex because it includes not only structured and complex relational information, but also semi-structured, heterogeneous and unstructured information. The 3 Vs of Big Data has been illustrated in figure 2.