Technologies for Big Data

Technologies for Big Data

Kapil Bakshi (Cisco Systems Inc., USA)
Copyright: © 2014 |Pages: 22
DOI: 10.4018/978-1-4666-4699-5.ch001
OnDemand PDF Download:


This chapter provides a review and analysis of several key Big Data technologies. Currently, there are many Big Data technologies in development and implementation; hence, a comprehensive review of all of these technologies is beyond the scope of this chapter. This chapter focuses on the most popularly accepted technologies. The key Big Data technologies to be discussed include: Map-Reduce, NOSQL technology, MPP (Massively Parallel Processing), and In Memory Databases technologies. For each of these Big Data technologies, the following subtopics are discussed: the history and genesis of the Big Data technologies, problem set that this technology solves for Big Data analytics, the details of the technologies, including components, technical architecture, and theory of operations. This is followed by technical operation and infrastructure (compute, storage, and network), design considerations, and performance benchmarks. Finally, this chapter provides an integrated approach to the above-mentioned Big Data technologies.
Chapter Preview

Introduction: The Challenge Of Big Data

The amount of data in the world is being collected and stored at unprecedented rates. A study by IDC Gantz & Reinsel, (2011) indicates that the world’s information is doubling every two years. Also the IDC study by Gantz & Reinsel (2011), mentions that the world created a staggering 1.8 zettabytes of information (a zettabyte is 1000 exabytes), and projections suggest that by 2020, we’ll generate will generate 50 times that amount.

Big Data has been defined as, when data sets get so large, that traditional technologies, techniques, and tools for extracting insights are no longer useful in a reasonable timeframe and cost-effective manner. This has spawned a new generation of technologies and corresponding considerations. Desai, Kommu & Rapp (2011) examine the cause of this explosion of Big Data, the following factors dominate:

  • Mobility trends: Mobile devices and sensor proliferation;

  • New data access: Internet, interconnected systems, and social networking;

  • Open source model: Major changes in the information processing model and the availability of an open source framework.

What distinguishes Big Data from data in the past, however, is not just its vast volume. The defining features of Big Data are also its variety—the sources and types of data being collected—and its velocity, the speed at which the data is flowing through the networked systems. Studies like Cisco Virtual Networking Index by Barnett, (2011) estimate that in 2016, global IP traffic will reach 1.3 zettabytes per year or 110.3 exabytes per month. Moreover, it is anticipated that there will be 19 billion networked devices by 2016.

One of the most interesting aspects about Big Data is that that unstructured data is the fastest growing type of data. Unstructured data refers to information that either does not have a predefined data model or does not fit well into relational database tables. Examples of unstructured data include imagery, sensor data, telemetry data, video, documents, log files, and email files. The challenge is not only to store and manage this vast mix, but to analyze and extract meaningful value from it—and to do so in a reasonable timeframe and at a reasonable cost.

Fortunately, a new generation of technologies has emerged for collecting, storing, processing, and analyzing Big Data. This chapter provides a survey of these technologies, including implementation and operational details:

  • MapReduce framework, including Hadoop Distributed File System (HDFS);

  • NoSQL (Not Only SQL) data stores;

  • MPP (Massively Parallel Processing) databases;

  • In-memory database processing.


Mapreduce And Hadoop Distributed File System

In 2003, Google published a paper on Google File Systems (GFS) by Ghemawat, Gobioff & Leung (2003). and a subsequent paper on the MapReduce Dean & Ghemawat (2004) model to address processing large unstructured data sets. Hadoop, an open source framework developed by the Apache Foundation, is an outgrowth of the concepts presented by Google in these papers. The Hadoop project has both a distributed file system, called HDFS (Hadoop Distributed File System) modeled on GFS and a distributed processing framework using MapReduce concepts. This chapter will review Big Data Analytics technologies with the Hadoop project as a backdrop.

To get a big-picture understanding of the technology innovations embodied in Hadoop, let’s start with the MapReduce programming model. The model, whose name comes from the map and reduces functions, uses a large number of computer nodes, connected via network interconnect, in a cluster fashion, to perform parallel processing across huge data sets.

Four characteristics —parallelism, fault tolerance, scalability, and data locality—are, in fact, the defining features of MapReduce systems, by White (2012):

Complete Chapter List

Search this Book:
Editorial Advisory Board and List of Reviewers
Table of Contents
Wen-Chang Fang
Wen-Chen Hu, Naima Kaabouch
Wen-Chen Hu, Naima Kaabouch
Chapter 1
Kapil Bakshi
This chapter provides a review and analysis of several key Big Data technologies. Currently, there are many Big Data technologies in development and... Sample PDF
Technologies for Big Data
Chapter 2
Ilias K. Savvas, Georgia N. Sofianidou, M-Tahar Kechadi
Big data refers to data sets whose size is beyond the capabilities of most current hardware and software technologies. The Apache Hadoop software... Sample PDF
Applying the K-Means Algorithm in Big Raw Data Sets with Hadoop and MapReduce
Chapter 3
Gueyoung Jung, Tridib Mukherjee
In the modern information era, the amount of data has exploded. Current trends further indicate exponential growth of data in the future. This... Sample PDF
Synchronizing Execution of Big Data in Distributed and Parallelized Environments
Chapter 4
Ahmet Artu Yıldırım, Cem Özdoğan, Dan Watson
Data reduction is perhaps the most critical component in retrieving information from big data (i.e., petascale-sized data) in many data-mining... Sample PDF
Parallel Data Reduction Techniques for Big Datasets
Chapter 5
Lynne M. Webb, Yuanxin Wang
The chapter reviews traditional sampling techniques and suggests adaptations relevant to big data studies of text downloaded from online media such... Sample PDF
Techniques for Sampling Online Text-Based Data Sets
Chapter 6
Francesco Di Tria, Ezio Lefons, Filippo Tangorra
Traditional data warehouse design methodologies are based on two opposite approaches. The one is data oriented and aims to realize the data... Sample PDF
Big Data Warehouse Automatic Design Methodology
Chapter 7
M. Asif Naeem, Gillian Dobbie, Gerald Weber
In order to make timely and effective decisions, businesses need the latest information from big data warehouse repositories. To keep these... Sample PDF
Big Data Management in the Context of Real-Time Data Warehousing
Chapter 8
Jeonghyun Kim
The goal of this chapter is to explore the practice of big data sharing among academics and issues related to this sharing. The first part of the... Sample PDF
Big Data Sharing Among Academics
Chapter 9
Chris A. Mattmann, Andrew Hart, Luca Cinquini, Joseph Lazio, Shakeh Khudikyan, Dayton Jones, Robert Preston, Thomas Bennett, Bryan Butler, David Harland, Brian Glendenning, Jeff Kern, James Robnett
Big data as a paradigm focuses on data volume, velocity, and on the number and complexity of various data formats and metadata, a set of information... Sample PDF
Scalable Data Mining, Archiving, and Big Data Management for the Next Generation Astronomical Telescopes
Chapter 10
Zorica Stanimirović, Stefan Mišković
This study presents a novel approach in analyzing big data from social networks based on optimization techniques for efficient exploration of... Sample PDF
Efficient Metaheuristic Approaches for Exploration of Online Social Networks
Chapter 11
Stacy T. Kowalczyk, Yiming Sun, Zong Peng, Beth Plale, Aaron Todd, Loretta Auvil, Craig Willis, Jiaan Zeng, Milinda Pathirage, Samitha Liyanage, Guangchen Ruan, J. Stephen Downie
Big Data in the humanities is a new phenomenon that is expected to revolutionize the process of humanities research. The HathiTrust Research Center... Sample PDF
Big Data at Scale for Digital Humanities: An Architecture for the HathiTrust Research Center
Chapter 12
Tanu Malik
Data-rich scientific disciplines increasingly need end-to-end systems that ingest large volumes of data, make it quickly available, and enable... Sample PDF
GeoBase: Indexing NetCDF Files for Large-Scale Data Analysis
Chapter 13
Joaquin Vanschoren, Ugo Vespier, Shengfa Miao, Marvin Meeng, Ricardo Cachucho, Arno Knobbe
Sensors are increasingly being used to monitor the world around us. They measure movements of structures such as bridges, windmills, and plane... Sample PDF
Large-Scale Sensor Network Analysis: Applications in Structural Health Monitoring
Chapter 14
Mian Lu, Qiong Luo
Large-scale Genome-Wide Association Studies (GWAS) are a Big Data application due to the great amount of data to process and high computation... Sample PDF
Accelerating Large-Scale Genome-Wide Association Studies with Graphics Processors
Chapter 15
Nathan Regola, David A. Cieslak, Nitesh V. Chawla
The selection of hardware to support big data systems is complex. Even defining the term “big data” is difficult. “Big data” can mean a large volume... Sample PDF
The Need to Consider Hardware Selection when Designing Big Data Applications Supported by Metadata
Chapter 16
Charles Loboz
Modern data centers house tens of thousands of servers in complex layouts. That requires sophisticated reporting – turning available terabytes of... Sample PDF
Excess Entropy in Computer Systems
Chapter 17
Raghunath Nambiar, Meikel Poess
Industry standard benchmarks have played, and continue to play, a crucial role in the advancement of the computing industry. Demands for them have... Sample PDF
A Review of System Benchmark Standards and a Look Ahead Towards an Industry Standard for Benchmarking Big Data Workloads
About the Contributors