How Big Does Big Data Need to Be?

Background

Sample size determination is a topic that has often been examined in medical or sociological science (e.g., Brutti et al. 2009, Sahu & Smith 2006, Santis 2007), since, in these fields, samples are often expensive in comparison to big data environments. Additionally, the available methods to determine the needed sample size often focus on a specific task, such as to find the needed number of participants of a survey. Therefore, these methods do not seem appropriate for a generally applicable framework in predictive analytics with its variety of machine learning and data mining techniques.

In contrast to the available methods that calculate the needed sample size a priori, the proposed process is based on the evaluation of the predictive accuracy and the calculation of the economic value of the classifier.

The predictive accuracy of classifiers with dichotomous outcomes can be calculated by integrating the receiver operator characteristic (ROC) curve. The obtained value is called the area under the curve (AUC; Bradley 1997), which represents the probability that a data record with unknown class is classified correctly. In the case study, the authors employ two logistic regression models with elastic net regularization (Friedman et al. 2010) to estimate the model parameters that the authors use to predict the dependent variable on the holdout sample. Based on these predictions, the authors show that increasing the sample size results in convergence the AUC. Other types of dependent variables, such as multinomial outcomes, require other measures, such as the misclassification error.