A Context-Based Performance Enhancement Algorithm for Columnar Storage in MapReduce with Hive

2. Background

To carry out data-intensive analysis in a scalable, fault-tolerant and efficient manner for a distributed environment, Google introduced a distributed and parallel programming framework called MapReduce (Condie et al., 2010; Dean & Ghemawat, 2010). The MapReduce framework is highly desirable as it allows a programmer to specify the analytical job and address the issue of translating the job into sub tasks on multiple machines which are completely automated. Under the umbrella of Apache, an open source implementation of the MapReduce framework, referred to as Hadoop, is freely available to both commercial and academic users. Given its easy access, Hadoop has become a popular choice to process big data produced by the web applications and business industry. Furthermore, due to the success of Hadoop and MapReduce, there is a significant interest in the traditional data warehousing industry to explore the integration of the MapReduce paradigm for large-scale analytical processing of relational data. The two major efforts to provide a declarative interface on top of Hadoop run-time environment are the Pig from Yahoo! and the Hive from Facebook. (Dittrich et al., 2012; Kaldewey et al., 2012; Pavlo et al., 2009; Songting et al., 2010; Thusoo et al., 2009)