Article Preview
Top1. Introduction
The data which is associated with characteristics like huge velocity, volume and variety is termed as “Big Data”. The expansion of the digital world is further accelerating has rate at which this data is generated. Gantz and Reinsel (2012) have mentioned that as per the IDC prognosis, the digital universe will reach a stupendous size in another year which will be 300 times its size around a decade ago. IBM Big Data and Analytics Hub has presented the fourth V, “Veracity”, which consolidates the vulnerability of information (IBM Big Data & Analytics Hub, “The Four V's of Big Data,” n.d.). With the ascent of such information, the conventional methods fail to meet the demands of real*world applications. The cloud computing technology which makes use of distributed architecture has thus emerged as the one of the prime innovations fulfilling the needs of execution, effectiveness and accessibility. The Apache based data processing platforms like Hadoop (Apache Hadoop, n.d.), Spark (Apache Spark, n.d.) and Storm (Apache Storm, n.d.) are nowadays widely used for processing the big data.
Hadoop data platform performs batch processing by making use of HDFS (Hadoop Distributed File System) and MapReduce (Map and Reduce functions) framework. In distributed environment of large clustered systems, HDFS is used for data storage and processing of data is done using the MapReduce framework (Apache Hadoop, n.d.). Spark data platform is an open source cluster-based framework which provides faster data processing than Hadoopp due to resilient distributed data-set architecture (Apache Spark, n.d.). Storm is a real-time distributed computation system used as a stream processing solution in large clusters (Apache Storm, n.d.).
These widely used data platforms have more than hundreds of configurable system parameters. These parameters are normally tuned to certain default values. The purpose of the research work carried here is to assess the role, the impact of these system parameters and the values to which they are tuned in defining the completion of a given job with improved efficiency. To induce the foremost of every system, it is essential to tune these parameters to the best possible values. In current scenario these values are set according to the instinct and experience of the service provider, subsequently which might not always lead to the most ideal setup for offering services especially in a model like pay-as-you-go.
Earlier work carried out by different researchers suggests two methods currently adopted to find the optimal configuration for a system offering platform-as-a-service (PaaS). The first method involves an exhaustive trial-and-error methodology in which several attempts are made to identify the “best value” for each parameter. This method is intrinsically infeasible looking at the example stated here. Let us assume that service administrator needs to try 10 values for each parameter; for a parameter set of 100 parameters. For this one has to empirically note observations of the magnitude 10100 making it practically exhaustive. The second approach is using machine learning techniques to find tailor-made parameter values for a system setup (Wang, Xu, & He, 2016) (Trotter, Liu, & Wood, 2017). The second approach is robust and flexible enough although this too involves vast computation.
After identifying the research gaps, this paper proposes the heuristic optimal values for the configurable parameters of Apache framework data platforms. The parameters tuned here are identified using filter and embedded approaches of feature selection techniques (Pattanshetti and Attar, in press). Applying the feature selection technique helped in identifying the reduced feature space for all three data platforms and in eliminating the not-so relevant and redundant features. The commonly identified features by various filter and embedded algorithms eventually made to the final feature space producing the optimal feature set. This optimal feature set is used for tuning, to empirically assess the impact of every input parameter on the output parameter of the respective data platform. The results show significant improved performance when these input parameters are tuned to the heuristic optimal values as compared to when assigned with default values.