This section introduces the rising of a new parallel computing technology for big data - MapReduce, and compares MapReduce against RDBMS (relational database management systems).
Rising of MapReduce
MapReduce was introduced by Google in 2004 (Dean & Ghemawat, 2004) to process big volume of unstructured data. Now it has become a standard tool for big data processing and analytics. In industry, dozens of big data startups are launched, building their businesses around the MapReduce technology. They are Cloudera, HortonWorks, MapR, Karmasphere, DataMeer, Aster Data, Greenplum, Hadapt, and Platfora etc. In academia, MapReduce has aroused tides of research in parallel computing and database community.
The research has touched almost every aspect of MapReduce (Lee, Lee, Choi, Chung, & Moon, 2011; Sakr, Liu, & Fayoumi, 2013), including: (1) Storage layout, data placement, handling of data skew, index support, and data variety support. (2) Extension of MapReduce for stream processing, incremental & continuous processing, iterative processing, leveraging of large memory of the cluster. (3) Optimization of joining, parallelization of complex analytical algorithms. (4) Schedule strategies for multi core CPU, GPU, heterogeneous environment, and cloud. (5) Easy to use interfaces for SQL, statistical algorithms, data mining & machine learning algorithms. (6) Energy saving, private and security guarantee. Due to space limitation, readers can refer to the two above mentioned references for more details. The references can be used as two hubs to recent research literatures.
Hadoop is an Apache project founded by Doug Cutting, and the Hadoop software stack is an open source implementation of the MapReduce technology. Throughout this paper, the two terms of MapReduce and Hadoop are used interchangeably. MapReduce is used when we express the general concept of MapReduce computing model (not specifically referring to Google’s MapReduce platform), and Hadoop is used when some products of vendors, which are based on Hadoop, are introduced.
Traditional players in the database market also noticed the popularity of MapReduce. IBM moves quickly with its Big Insights Plan, which tries to integrate DB2, Hadoop, Netezza, and SPSS into a big data analytic platform. TeraData acquired Aster Data to obtain its experience of MapReduce as well as the analytic software package using MapReduce-style parallelization. EMC, formerly not as a database vendor, became a big player in the market overnight through acquiring Greenplum. Several venders who looked down on MapReduce before finally change their minds, Microsoft rejected MapReduce in 2009, and in 2012 it has closed the Dryad project (Foley, 2011) (a parallel computing framework similar to MapReduce) and warmly hugs Hadoop. Oracle despised MapReduce in early 2011, finally published its Big Plan which involved noSQL/Hadoop providing in late 2011.