Understanding Big Data

Introduction

Big data has been one of the major areas of focus in the field of data management. Big data provides the business solutions which help the organizations making their decisions. Current growing value for the data helps organizations innovate quickly the optimum usage of data and keep up the edge (Lukoinova and Rubin, 2014).

Implementation of methodologies should be in context with a technology base that is growing to be a moving target. The main technology behind fostering the rate of innovation in big data platforms and solutions is the open source technology development and delivery model. Organizations face challenges with evolving business needs and technologies, organizations hold the flexibility for the platforms, solutions, and evolving their capabilities so that they derive value and positive insights from their big data investments (Nimmagadda and Dreher, 2013).

According to the latest Worldwide Semiannual Big Data and Analytics Spending Guide from International Data Corporation (IDC), worldwide revenues for big data and business analytics (BDA) will grow from $130.1 billion in 2016 to more than $203 billion in 2020 (IDC, 2015).

Figure 1.

Market predictions on big data (USD Billion)

Source: IDC (2015)

Organizations which handle the big data and implement its methodologies are expected to make 40% more profits than regular software industry does in the current scenario. The increasing value for big data makes it easier to predict the gains for the organization in the future. Organizations currently lack the human resource and talent which can give them the best big data engineering experience and help them grow.

The era of big data has established a new path for exploring data in newer forms and finding different ways to handle the data on a large scale. Although processing and maintaining a large data is a challenge, big data challenges have given the scope to find a solution for these challenges and implement them for a better data environment (Chen et al., 2013). Big data has been into existence since the 1990s and data integration has been one of the major challenges since then. Data Integration in large: Challenges of Reuse, a research paper which was published in 1994 signifies the existence of big data from 1990s.

This chapter defines Big Data basically and provides an overview of Big Data in terms of current status, organizational effects (technology, health care, education etc.), implementation challenges and Big Data projects. This research adopted literature review as methodology and refined valuable information through current journals, books, magazines and blogs.

Background

Evolution of large data sets from major industries is termed as big data in the field of data science. The first large-scale methods for metadata creation and analysis (an arrangement of clay tablets revealing data about livestock) has been linked to the Sumerian people, active in the early Bronze Age (Erikson, 1950). Similarly, card catalogs, and other information methods used in libraries (Lee, Clarke and Perti, 2015), are forerunners to the large-scale digitized metadata collections of today, as they too were technologies used for gathering and storing facts about data in comprehensive and systematized ways (Lee, Clarke and Perti, 2015). The rise in digital technology is leading to the overflow of data (Gog et al., 2015), which constantly requires more updated and faster data storage systems (Sookhak, Gani, Khan, and Buyya, 2017). The recognition of data excess started as early as the 1930s but was not actually named Big Data until the mid-1990s by John Mashey (Kitchin and McArdle, 2016). The sudden increase in the U.S. population, the dispensing of social security numbers, and the wide-ranging increase of knowledge (research) required more detailed and organized record-keeping (Gandomi and Haider, 2015).