Selection of the Best Subset of Variables in Regression and Time Series Models

Introduction

Variable selection refers to the problem of selecting input variables that are most predictive of a given outcome. Variable selection problems are found in all supervised or unsupervised machine learning tasks, classification, regression, time series prediction, pattern recognition.

In the recent years, variable selection has become the focus of considerable research in several areas of application for which datasets with tens or hundreds of thousands of variables are available. These areas include text processing, particularly in application to Internet documents, and genomics, particularly gene expression array data. The objective of variable selection is three-fold: to improve the prediction performance of the predictors, to provide faster and more cost-effective predictors, and to provide a better understanding of the underlying process that generated the data.

A number of studies in the statistical literature discuss the problem of selecting the best subset of predictor variables in regression. Such studies focus on subset selection methodologies, selection criteria, or a combination of both. The traditional selection methodologies can be enumerative (e.g. all subsets and best subsets procedures), sequential (e.g. forward selection, backward elimination, stepwise regression, and stagewise regression procedures), and screening-based (e.g. ridge regression and principal components analysis). Standard texts like Draper and Smith (1981) and Montgomery and Peck (1992) provide clear descriptions of these methodologies.

Some of the reasons for using only a subset of the available predictor variables (given by Miller, 2002) are:

• • To estimate or predict at a lower cost by reducing the number of variables on which data are to be collected;

• • To predict more accurately by eliminating uninformative variables;

• • To describe multivariate data sets parsimoniously; and

• • To estimate regression coefficients with smaller standard errors (particularly when some of the predictors are highly correlated).

These objectives are of course not completely compatible. Prediction is probably the most common objective, and here the range of values of the predictor variables for which predictions will be required is important. The subset of variables giving the best predictions in some sense, averaged over the region covered by the calibration data, may be very inferior to other subsets for extrapolation beyond this region. For prediction purposes, the regression coefficients are not the primary objective, and poorly estimated coefficients can sometimes yield acceptable predictions. On the other hand, if process control is the objective then it is of vital importance to know accurately how much change can be expected when one of the predictors changes or is changed.

Suppose that y, a variable of interest, and x₁, ..., x_v, a set of potential explanatory variables or predictors, are vectors of n observations. The problem of variable selection, or subset selection as it is often called, arises when one wants to model the relationship between y and a subset of x₁, ..., x_v, but there is uncertainty about which subset to use. Such a situation is particularly of interest when v is large and x₁, ..., x_v is thought to contain many redundant or irrelevant variables.

The variable selection problem is most familiar in the linear regression context, where attention is restricted to normal linear models. Letting w index the subsets of x₁, ..., x_v and letting p_w be the number of the parameters of the model based on the wth subset, the problem is to select and fit a model of the formy = X_wa_w + ε, (1) where X_w is an n × p_w matrix whose columns correspond to the wth subset, a_w is a p_w × 1 vector of regression coefficients, and ε~N_n(0,σ²I). More generally, the variable selection problem is a special case of the model selection problem where each model under consideration corresponds to a distinct subset of x₁, ..., x_v. Typically, a single model class is simply applied to all possible subsets.