Adapting Big Data Ecosystem for Landscape of Real World Applications

Adapting Big Data Ecosystem for Landscape of Real World Applications

Jyotsna Talreja Wassan (Maitreyi College, University of Delhi, India)
Copyright: © 2018 |Pages: 12
DOI: 10.4018/978-1-5225-2255-3.ch029
OnDemand PDF Download:
$30.00
List Price: $37.50

Abstract

Big data is revolutionizing world in the age of Internet. The wide variety of areas like online businesses, electronic health management, social networking, demographics, geographic information systems, online education etc. are gaining insight from big data principles. Big data is comprised of heterogeneous datasets which are too large to be handled by traditional relational database systems. An important reason for explosion of interest in big data is that it has become cheap to store volumes of data and there is a major rise in computation capacity. This article aims at giving an overview of Big Data Ecosystem comprising various big data platforms useful in today's competitive world.
Chapter Preview
Top

Introduction

Big data is revolutionizing world in the age of Internet. The wide variety of areas like online businesses, electronic health management, social networking, demographics, geographic information systems, online education etc. are gaining insight from big data principles. Big data is comprised of heterogeneous datasets which are too large to be handled by traditional relational database systems. An important reason for explosion of interest in big data is that it has become cheap to store volumes of data and there is a major rise in computation capacity.

To extract valuable patterns from big data, one needs to choose a right platform for capturing, organizing, searching and analyzing the context of voluminous data.

Data Management sytems adhering to big data, aim to add computing nodes to cater to increasing data volumes, automatic balancing of data between various nodes, and reducing the operational cost for functioning of distributing data over various nodes (Patel, 2016).

Various NoSQL data stores like Cassandra, MongoDB and Hadoop HBASE etc. are in use today to acquire, manage, store and query big data. NoSQL databases are inherently schema-less and permit records to have variable number of fields, making them distinct from other non-relational databases like hierarchical databases and object-oriented databases. These are highly scalable and well suited for dynamic data structures. NoSQL data is characterized by being basically available and eventually consistent.

The frameworks like MapReduce, Dryad etc. support processing of large amounts of data in parallel and hence the management of big data (Singh & Reddy, 2015). The technologies like GNU R and Apache MAHOUT are useful in exploring and analyzing big data for finding relevant valuable patterns. This article aims at giving an overview of Big Data Ecosystem comprising various big data platforms useful in today’s competitive world.

Top

Background

In 1970’s big meant megabytes, subsequently with the increasing data needs, it grew to gigabytes and terabytes and further to zettabytes with the increase in digital information. The traditional world of relational database systems like Oracle RDBMS etc. faced challenges in storing large quantities of data and needed to scale databases to data volumes beyond the storage and/or processing capabilities of a single large computer system. Many efforts have been made to store and manage data being generated from everywhere on the web. Several database management systems were proposed on the basis of master/slave, cluster computing or partitioning architecture like IBM DB2 partitioning, VoltTB etc.

However, the problems in reliance on shared facilities and resources (CPU, Disk, and Processors), scalability and complex administration limitations, augmented by lack of support for critical requirements, led to development of SHARED NOTHING architectures (Strauch, 2011; Lee, 2011) in 1980’s. These systems focused on parallel and distributed data computation and solved big data problems using parallel computations. By 90’s, even these solutions faced challenges in running OLTP and queries due to data overload. To provide solutions to these problems, Google responded with its GFS (Dean & Ghemawat, 2004), followed by a powerful programming paradigm of MapReduce (Dean & Ghemawat, 2004) .Thereafter a spectrum of new technologies emerged as the NoSQL movement stating a broad class of database management system to support increasing data storage and analytical requirements.

Key Terms in this Chapter

Online Transaction Processing (OLTP): It refers to a class of systems that facilitate and manage transaction-oriented applications, typically for data entry and retrieval transaction processing .

Database Management Systems (DBMS): A Database Management Systems (DBMS) is a set of programs that enables storing, adding, deleting, accessing, modifying, updating or analyzing data stored in one location.

Distributed System: It consists of autonomous machine nodes connected in a network to communicate, share and coordinate their activities through message passing to achieve a common goal.

Data Intensive Computing: It classify various parallel computing applications which use a data parallel approach to process large volumes of data like terabytes or petabytes in size and referred to as Big Data.

Big Data: A term that describes large amount of data that could be structured, unstructured or real time and is difficult to be handled by traditional systems.It is identified by major 3 V’s: Volume, Variety and Velocity.

MapReduce: MapReduce is a parallel programming model proposed by Google and is used to distribute computing on clusters of computers for processing large data sets.

Complete Chapter List

Search this Book:
Reset