Article Preview
Top1. Introduction
Artificial neural networks (ANNs) are known as universal approximators and computational models with remarkable properties such as adaptability, capacity of learning by examples and the ability to generalize data (Haykin, 1998).
Neural networks are of great use in pattern classification applications, and through a supervised learning perspective, they are considered a general method for constructing mappings between a group of sample vectors (training set) and the corresponding classes, allowing the classification of unseen data as one of the classes learned in the training process.
One of the most used ANN models is the well-known Multi-Layer Perceptron (MLP). The training process of MLPs for pattern classification consists of two main tasks: selection of an appropriate architecture for the problem and the adjustment of the connection weights of the network.
Gradient-based learning strategies, such as Backpropagation (BP) and its variant Levenberg-Marquardt (LM-BP), have been extensively used in the training of MLPs, but these approaches are usually slower than required in learning, and may also get stuck in local minima (Zhu et al., 2005).
Extreme learning machine (ELM) was proposed as an efficient learning algorithm for single-hidden layer feedforward neural network (SLFN) (Huang et al., 2006). ELM increases the learning speed by means of randomly generating weights and biases for hidden nodes, differently from gradient-based approaches, which commonly tune iteratively the network parameters.
Although ELM is fast and presents good generalization performance, as the output weights are computed based on the prefixed input weights and hidden biases using the Moore-Penrose (MP) generalized inverse, there may exist a set of non-optimal input weights and hidden biases, and it might suffer from the overfitting as the learning model will approximate all training samples well.
Global search techniques, such as Tabu Search (TS) (Glover, 1986), Evolutionary Algorithms (EAs, like Genetic Algorithm - GA) (Eiben & Smith, 2003), Differential Evolution (DE) (Storn & Price, 1995; Storn & Price, 1997), Particle Swarm Optimization (PSO) (Kennedy & Eberhart, 1995; Kennedy & Eberhart, 2001; van den Bergh, 2002) and Group Search Optimization (GSO) (He et al., 2006; He et al., 2009), are widely used in scientific and engineering problems, and these strategies have been combined with ANNs to perform various tasks, such as connection weight initialization, connection weight training and architecture design.
In this paper, we present two new hybrid evolutionary approaches based on Particle Swarm Optimization technique to select input weights and hidden biases for Extreme Learning Machine neural network: PSO-ELM-CS2 and GCPSO-CS2. These methods are extensions from the PSO-ELM-CS1 and GCPSO-ELM-CS1 approaches, respectively, presented in Pacifico and Ludermir (2012). The Particle Swarm Optimization (PSO) consists of a stochastic global search originated from the attempt to graphically simulate the social behavior of a flock of birds looking for resources.
For the proposed methods, individuals in the PSO population were divided into groups by a clustering algorithm, following the idea of “population stereotyping” presented in Kennedy (2000). The lbest topology was adopted in a way that PSO particles will update according to individuals from its neighborhood. A selection operator was also applied to all strategies, based on the ideas of Angeline (1999).
Some evolutionary strategies have been adopted for the ELM context. Zhu et al. (2006) introduces a hybrid form of differential evolutionary (DE) algorithm to search for optimal input weights and hidden biases for ELM, called E-ELM to train SLFN with more compact networks.