Statistical Study of Machine Learning Algorithms Using Parametric and Non-Parametric Tests: A Comparative Analysis and Recommendations

Statistical Study of Machine Learning Algorithms Using Parametric and Non-Parametric Tests: A Comparative Analysis and Recommendations

Vijay M. Khadse (College of Engineering Pune, (CoEP), Pune, India), Parikshit Narendra Mahalle (Department of Artificial Intelligence and Data Science, Bansilal Ramnath Agarwal Charitable Trust's, Vishwakarma Institute of Information Technology, India) and Gitanjali R. Shinde (Smt. Kashibai Navale College of Engineering, Pune, India)
Copyright: © 2020 |Pages: 26
DOI: 10.4018/IJACI.2020070105
OnDemand PDF Download:
No Current Special Offers


The emerging area of the internet of things (IoT) generates a large amount of data from IoT applications such as health care, smart cities, etc. This data needs to be analyzed in order to derive useful inferences. Machine learning (ML) plays a significant role in analyzing such data. It becomes difficult to select optimal algorithm from the available set of algorithms/classifiers to obtain best results. The performance of algorithms differs when applied to datasets from different application domains. In learning, it is difficult to understand if the difference in performance is real or due to random variation in test data, training data, or internal randomness of the learning algorithms. This study takes into account these issues during a comparison of ML algorithms for binary and multivariate classification. It helps in providing guidelines for statistical validation of results. The results obtained show that the performance measure of accuracy for one algorithm differs by critical difference (CD) than others over binary and multivariate datasets obtained from different application domains.
Article Preview


Ambient computing provides an environment of response which helps businesses to perform to full capacity, helps to remove many intermediate processing steps with its ability to collect data to perform analytics and learn from it. At the core of ambient computing is IoT. Most of the developed countries use Internet of Things (IoT) to provide better services to their people e.g. smart transport, smart health and smart energy. The idea is to make use of internet-enabled devices without deliberately or intentionally using them. It is about moving computation from foreground to background. It assumes shifting of computing from desktop to invisible smart computing devices incorporated into human environment. These devices interact with humans through sensing or recognizing human activities using different technologies.

One of the requirements of ambient computing is to learn and adapt to users context by observing human activities. To deliver self-adjusting services, it is important and essential to collect background information using sensors and to use this accumulated data for drawing inferences and classification of situations. Machine learning methods enable ambient computing system to learn in a highly dynamic environment, thus making it more convenient, efficient and acceptable to the user.

Over the years, machine learning community has been facing a challenge of identifying the best learning algorithm to be applied to data from different application domains e.g. language processing, cyber security, agriculture, transportation, smart cities, medical imaging, remote sensing, software engineering, manufacturing, etc., with different dimensions and attributes, to learn and maximize. Many real-world applications in the domain of language processing, medical imaging, cyber security use machine learning (ML) techniques to enhance their efficiency and applicability (Dey, Wagh, Mahalle, & Pathan, 2019). Due to availability of open machine learning frameworks, it is relatively easy to develop a new machine learning algorithm or to modify the existing algorithm. A newly developed or modified algorithm needs to be compared with existing algorithms in terms of their performances. The results obtained need to statistically be validated using appropriate tests. Additionally, there is no algorithm that is fittest for all the problems and hence the challenge is to identify an optimal algorithm for a specific type of predictive analysis task (Dietterich, 1998). There are many elements in machine learning that influence the decision to select the best machine learning algorithmic technique. Many researchers present reports using simplistic and known statistical parameters like mean and variance. Comparison of ML algorithms on problems from different domain lacks in significance of accuracy obtained in the results. Therefore, statistical validation of the results obtained is extremely important (Demšar, 2006).

A remarkable number of researchers used statistical techniques for validation, but the size of datasets was smaller than the real-world datasets. The number of datasets used in the study was comparatively less, which highly affected the performance and gave incorrect results in many known cases (Luengo, Garcia, & Herrera, 2009). In remote sensing kind of applications, comparison is done between pixel based and object-based image analyses using a conventional parametric classification scheme and non-parametric schemes for the classification of agriculture landscaped (Duro, Franklin, & Dube, 2012). The intension behind a multiple comparison analysis is to test a new algorithm in comparison with the existing algorithms (Wu, Huang & Meng, 2008). In the past, very few publications validated their results using appropriate statistical tests. Nevertheless, those who used it mostly used a parametric test. It has been observed that some studies used parameters such as variance and mean. Some of the studies used t-test or ANOVA. (Zekic-Susac & Horvvat, 2005; Li, Shiue & Huang, 2006; Kim, 2008).

Complete Article List

Search this Journal:
Volume 14: 1 Issue (2023): Forthcoming, Available for Pre-Order
Volume 13: 6 Issues (2022): 1 Released, 5 Forthcoming
Volume 12: 4 Issues (2021)
Volume 11: 4 Issues (2020)
Volume 10: 4 Issues (2019)
Volume 9: 4 Issues (2018)
Volume 8: 4 Issues (2017)
Volume 7: 2 Issues (2016)
Volume 6: 2 Issues (2014)
Volume 5: 4 Issues (2013)
Volume 4: 4 Issues (2012)
Volume 3: 4 Issues (2011)
Volume 2: 4 Issues (2010)
Volume 1: 4 Issues (2009)
View Complete Journal Contents Listing