An Analytical Approach for Optimizing the Performance of Hadoop Map Reduce Over RoCE

An Analytical Approach for Optimizing the Performance of Hadoop Map Reduce Over RoCE

Geetha J. (M.S. Ramaih Institute of Technology, Bengaluru, India), Uday Bhaskar N (Government College (UG & PG), Anantapur, India) and Chenna Reddy P. (JNTU-Anantapur, Anatapur, India)
DOI: 10.4018/IJICTHD.2018040101

Abstract

Data intensive systems aim to efficiently process “big” data. Several data processing engines have evolved over past decade. These data processing engines are modeled around the MapReduce paradigm. This article explores Hadoop's MapReduce engine and propose techniques to obtain a higher level of optimization by borrowing concepts from the world of High Performance Computing. Consequently, power consumed and heat generated is lowered. This article designs a system with a pipelined dataflow in contrast to the existing unregulated “bursty” flow of network traffic, the ability to carry out both Map and Reduce tasks in parallel, and a system which incorporates modern high-performance computing concepts using Remote Direct Memory Access (RDMA). To establish the claim of an increased performance measure of the proposed system, the authors provide an algorithm for RoCE enabled MapReduce and a mathematical derivation contrasting the runtime of vanilla Hadoop. This article proves mathematically, that the proposed system functions 1.67 times faster than the vanilla version of Hadoop.
Article Preview

Literature Review

As a prerequisite to our implementation, we investigate the existing High-Speed Interconnects on the bases of the scalability and performance. We discuss the potential of Remote Direct Memory Access and its applicability to MapReduce.

Complete Article List

Search this Journal:
Reset
Open Access Articles: Forthcoming
Volume 12: 4 Issues (2020): Forthcoming, Available for Pre-Order
Volume 11: 4 Issues (2019): 1 Released, 3 Forthcoming
Volume 10: 4 Issues (2018)
Volume 9: 4 Issues (2017)
Volume 8: 4 Issues (2016)
Volume 7: 4 Issues (2015)
Volume 6: 4 Issues (2014)
Volume 5: 4 Issues (2013)
Volume 4: 4 Issues (2012)
Volume 3: 4 Issues (2011)
Volume 2: 4 Issues (2010)
Volume 1: 4 Issues (2009)
View Complete Journal Contents Listing