Article Preview
TopIntroduction
Variable selection is a classical topic in regression which has many applications in several areas including, but not limited to, engineering, medicine, psychology, or business.
Among numerous variable selection methods developed, some classical sequential methods such as stepwise selection methods (Desboulets, 2018; Lindsey and Sheather, 2010) have been widely used because they are simple and work very well if there are not too many variables and they have low prediction error. But there are some drawbacks in these methods. Two most serious issues among them are (1) they tend to converge to local optima (Hans et al., 2012; Hocking,1976; Kiezun et al., 2009; Meiri and Zahavi, 2006; Paterini and Minerva, 2010) and (2) they do not work very well in high dimensional spaces. (Hand et al., 2012; Kapetanios, 2007). Later in this section, it will be explained how these problems can be resolved with ‘metaheuristics’ in optimization research.
The selection of the most adequate variables in regression models can be stated as a combinatorial optimization problem with the objective to select explanatory variables that maximize the adequacy of the model according to statistical criteria (objective function). (Meiri, 2006; Paterlini and Minerva, 2010) Some methods or algorithms from optimization research have been used for variable selection, including but not limited to, genetic algorithm (Broadhurst et al., 1997; Kapetanios, 2007; Kiezun et al., 2009; Jirapech-Umpai and Aitken, 2005; Mohan et al., 2018; Paterini and Minerva, 2010; Peng et al., 2005; Sinha et al., 2015), simulated annealing (Kiezun et al., 2009; Meiri and Zahavi, 2006), iterated local search (Hans et al., 2012). These methods are characterized as metaheuristics, a stochastic search strategy dedicated to solving difficult problems (NP-hard problems) in optimization research.
In particular, genetic algorithms (GA hereafter) and simulated annealing (SA hereafter) are known to be very effective to resolve the two issues mentioned above – (1) convergence to local optima (Kapatenios, 2007; Kiezun et al., 2006; Meiri, 2006; Paterini and Minerva, 2010) and (2) handling high dimensional spaces. (Kapatenious, 2007; Meiri, 2006) Brief descriptions of GA and SA can be found in Appendix.