Technologies for Handling Big Data

Introduction

In today’s world, every time we connect phone to internet, pass through a CCTV camera, order pizza online or even pay with credit card to buy some clothes, we generate data and that “Ocean of Data” is popularly known as Big Data. The amount of data that’s being created and stored on a universal level is almost inconceivable, and it just keeps growing. The amount of data we create is doubled every year. Big Data is a critical concept that integrates all kinds of data and plays an important role for strategic intelligence for any modern company. The importance of big data doesn’t revolve around how much data you have, but what you do with it. Big data is now the key for competition and growth for new startups, medium and big enterprises. Scientific research is now on boom using big data. For the astronomers, Sloan Digital Sky Survey has become a central resource. Big data has the potential to revolutionize research and education as well. A commonly quoted axiom is that “big data is for machines; small data is for people.” Big data is in its initial phase now and much more is to be discovered yet. The voluminous, varied and scattered data cannot be handled by traditional approaches and techniques and it prompted the development of various technologies which are required for handling big data. These technologies help the businesses (Pathak and Agrawal, 2019) and organizations (Agrawal and Gupta, 2017) for their specific and varied purposes. The future of big data can be imagined like the central nervous system of the planet. NoSQL is a non-relational database management system and designed for distributed data stores like Google or Facebook which collects terabits of data every day for their users. In these cases, storing data in fixed schema may not provide join and horizontal scalability.

In today’s moment data access and capturing through other parties is much easier such as Facebook, Google+ and others. Data has been increasing exponentially in some scenarios like Personal user information, social graphs, geo location data, user-generated content and machine logging (Agrawal and Gupta, 2018). To run these services in order, processing of huge data is required which SQL database can’t handle hence evolution of NoSQL databases happened. As organizations have become more familiar with the capabilities of big data analytics solutions, they have begun demanding faster and faster access to insights. For these enterprises, streaming analytics with the ability to analyze data as it is being created, is something of a holy grail. They are looking for solutions that can accept input from multiple disparate sources, process it and return insights immediately — or as close to it as possible. This is particular desirable when it comes to new IoT deployments, which are helping to drive the interest in streaming big data analytics.

The aim of this chapter is to discuss the technologies which are pertinent and essential for the big data. The important technologies for big data are-

• 1.

  Distributed and Parallel Computing

• 2.

  Schema less Databases

• 3.

  Map Reduce

• 4.

  Hadoop

• 5.

  Cloud Computing

• 6.

  Artificial Intelligence

• 7.

  Edge Computing

• 8.

  Blockchain

Dimensions Of Big Data

The general accord of the day is that there are explicit attributes available to define big data in an explicit manner. Generally, in most of big data circles, these are called the four V’s: volume, variety, velocity, and veracity.

Big Data has a lot of prospective for organizations and hence almost all big and middle level organizations are continuously looking to find the ways to apply its techniques in their organizations. For accelerating Big Data initiatives, Capgemini and Cloudera have announced an extended partnership recently. To smear this initiative, they have formed an infographic which outline five dimensions to define and characterize Big Data. Some of the important dimensions of big data are described here-