Multiple Decisional Query Optimization in Big Data Warehouse

Multiple Decisional Query Optimization in Big Data Warehouse

Ratsimbazafy Rado (University of Lyon, Bron, France) and Omar Boussaid (University of Lyon, Bron, France)
Copyright: © 2018 |Pages: 22
DOI: 10.4018/IJDWM.2018070102
OnDemand PDF Download:
No Current Special Offers


Data warehousing (DW) area has always motivated a plethora of hard optimization problem that cannot be solved in polynomial time. Those optimization problems are more complex and interesting when it comes to multiple OLAP queries. In this article, the authors explore the potential of distributed environment for an established data warehouse, database-related optimization problem, the problem of Multiple Query Optimization (MQO). In traditional DW materializing views is an optimization technic to solve such problem by storing pre-computed join or frequently asked queries. In this era of big data this kind of view materialization is not suitable due to the data size. In this article, the authors tackle the problem of MQO on distributed DW by using a multiple, small, shared and easy to maintain shared data. The evaluation shows that, compared to available default execution engine, the authors' approach consumes on average 20% less memory in the Map-scan task and it is 12% faster regarding the execution time of interactive and reporting queries from TPC-DS.
Article Preview


The business intelligence system has evolved since the growing interest in big data, and therefore data warehouses and OLAP (Online Analytical Processing) (Chaudhuri et al. (2011)). New models of data warehousing (DW) have appeared (Figure 1). We can cite, as examples: (1) DW based on relational database management systems (RDBMS), for example Teradata1, Greenplum2; and (2) DW based on MapReduce paradigm (Dean and Ghemawat, 2004) like Hadoop/Hive (Thusoo et al., 2010a).

Inside a big data warehouse system (Hadoop/Hive) there is still a physical independence between the data warehouse structure and data storage, as we have seen in RDBMS. The main difference relies on data storage. In Hadoop/Hive, data is stored following Hadoop Distributed File System (HDFS) method (block files, data replication, etc.). Nonetheless, there is a convergence between Hadoop/Hive and traditional RDBMS in how to retrieve information from the file system with SQL (Structured Query Language). We only specify what information we need, and the system will compute it to obtain the data from the storage unit. In Hadoop, a MapReduce framework is used to retrieve all data from HDFS and compute them to give the query result.

Since the first version of Hadoop/Hive, it has evolved to become more convenient to use for companies and researchers, but there are still several challenges to tackle, in the big data warehouse and OLAP (Cuzzocrea et al., 2013). One of those challenges is the end-user performance in query execution, particularly in case of multiple-query or intensive-query execution. With the constantly increasing amount of data, we are also facing a growing number of users. In this paper, our primary concern is the number of concurrent queries from users. The problem is known, in RDBMS, as multiple query optimization, or MQO (Sellis, 1988).

In this work, we are interested in the query execution in DW that uses MapReduce framework, particularly on multiple-query execution with MapReduce. As far as we know, in MapReduce framework, all queries are processed in parallel with Map and Reduce tasks and distributed on a cluster of computers. To execute a single query, MapReduce evaluates the query and generates the list of Map and Reduce tasks (Figure 3). During multiple-query execution MapReduce evaluates all queries one at a time, producing an execution latency and an enormous resource consumption, especially due to multiple read/write operations on HDFS distributed file system.

More research has occurred in the multiple-query processing in a distributed environment. Researchers have conducted diversified studies in: (1) system and resources tuning (Herodotou (2011), Herodotou (2012), Marcus (2016)), (2) job scheduling (Zaharia (2009), Isard (2009)) and (3) tasks and/or data sharing (Wang, 2013; Nykiel, 2010; Elgandour, 2012). Those works have pointed out that changing the cluster configuration, ordering the job execution or reusing intermediate results can improve in a significant way MapReduce efficiency during multiple-query or job execution.

Complete Article List

Search this Journal:
Volume 18: 4 Issues (2022): 2 Released, 2 Forthcoming
Volume 17: 4 Issues (2021)
Volume 16: 4 Issues (2020)
Volume 15: 4 Issues (2019)
Volume 14: 4 Issues (2018)
Volume 13: 4 Issues (2017)
Volume 12: 4 Issues (2016)
Volume 11: 4 Issues (2015)
Volume 10: 4 Issues (2014)
Volume 9: 4 Issues (2013)
Volume 8: 4 Issues (2012)
Volume 7: 4 Issues (2011)
Volume 6: 4 Issues (2010)
Volume 5: 4 Issues (2009)
Volume 4: 4 Issues (2008)
Volume 3: 4 Issues (2007)
Volume 2: 4 Issues (2006)
Volume 1: 4 Issues (2005)
View Complete Journal Contents Listing