Ensemble Learning for Regression

Background

Ensemble learning consists of two issues that need to be addressed, ensemble generation: how does one generate the base models/members of the ensemble and how large should the ensemble size be and ensemble integration: how does one integrate the base models’ predictions to improve performance? Some ensemble schemes address these issues separately, others such as Bagging (Breiman, 1996a) and Boosting (Freund & Schapire, 1996) do not. The problem of ensemble generation where each base learning model uses the same learning algorithm (homogeneous learning) is generally addressed by a number of different techniques: using different samples of the training data or feature subsets for each base model or alternatively, if the learning method has a set of learning parameters, these may be adjusted to have different values for each of the base models. An alternative generation approach is to build the models from a set of different learning algorithms (heterogeneous learning). There has been less research in this latter area due to the increased complexity of effectively combining models derived from different algorithms. Ensemble integration can be addressed by either one of two mechanisms: either the predictions of the base models are combined in some fashion during the application phase to give an ensemble prediction (combination approach) or the prediction of one base model is selected according to some criteria to form the final prediction (selection approach). Both selection and combination can be either static in approach, where the learned model does not alter, or dynamic in approach, where the prediction strategy is adjusted for each test instance.