In this case, we provide some basic information concerning the statistical methods used throughout this text. These methods include visualization by kernel density estimation, regression and predictive models, time series, and text analysis. This case is not intended to provide extensive information concerning these techniques; references are provided for the interested reader who wants to follow up on the methodology. Some of the methodology is fairly well known while the predictive modeling and text analysis are relatively recent. One of the primary visualization techniques used is that of kernel density estimation. Since the patient base is relatively heterogeneous, we cannot assume that any health outcome is normally distributed. Kernel density estimation can find the entire population distribution so that we no longer have to rely upon averages and can relate the results more to individuals.
TopData Mining Techniques
There are many different methods that can be used to investigate the data. We will discuss them briefly here. For more information, we refer the reader to Cerrito. (Cerrito, 2009a) These techniques include data visualization, predictive modeling, market basket analysis, time series analysis, and text analysis. Data mining is a process as well as a collection of techniques. We will list some of the differences between data mining and traditional statistical techniques.
Data mining deals with heterogeneous data, sometimes with a complex internal structure such as multimedia; including images, video, and text. Because most data collected in healthcare related to the routine of patient treatment consists of heterogeneous populations, the techniques of data mining are ideal to use with the various healthcare datasets. Data mining starts with the assumption that the raw data set is not of sufficient quality to apply a statistical model directly without some appropriate preprocessing techniques such that the preprocessing will have as much or even more influence on the quality of the final results compared to the selected statistical technique. Data mining uses flexible predictive techniques that often are based on algorithmic foundations, but may have weak formal statistical justification. The data mining process often uses hidden variables as tools to perform a step-by-step compression of raw input data. Data mining attempts to find not only general, global models based on a data set, but also to find basic patterns in the data. Data mining is more concentrated on the aspects of data management and optimization of a search through the data while statistics is more oriented toward formalisms for final model representation, and score function formalization in the data space to perform inference. Data mining has been more focused on estimation, and the process generally ignores inferential models.
Data mining and statistics have generally developed in different domains. Statisticians are primarily interested in inference; data miners in exploratory data analysis. Nevertheless, there are some instances where data mining and statistics have blended. Many statisticians remain dubious about the data mining process. (Lee, 1995) Others are concerned with the lack of a theoretical framework similar to the one for inferential statistics, especially since data mining tends to be algorithmic-based. (Giudiei & Passerone, 2002; Hand & Bolton, 2004; Sargan, 2001)