This chapter aims to further explore the capabilities of the Higher Order Neural Networks class and especially the Pi-Sigma Neural Networks. The performance of Pi-Sigma Networks is evaluated through several well known neural network training benchmarks. In the experiments reported here, Distributed Evolutionary Algorithms are implemented for Pi-Sigma neural networks training. More specifically, the distributed versions of the Differential Evolution and the Particle Swarm Optimization algorithms have been employed. To this end, each processor of a distributed computing environment is assigned a subpopulation of potential solutions. The subpopulations are independently evolved in parallel and occasional migration is allowed to facilitate the cooperation between them. The novelty of the proposed approach is that it is applied to train Pi-Sigma networks using threshold activation functions, while the weights and biases were confined in a narrow band of integers (constrained in the range [-32, 32]). Thus, the trained Pi-Sigma neural networks can be represented by using only 6 bits. Such networks are better suited for hardware implementation than the real weight ones and this property is very important in real-life applications. Experimental results suggest that the proposed training process is fast, stable and reliable and the distributed trained Pi-Sigma networks exhibit good generalization capabilities.
TopIntroduction
Evolutionary Algorithms (EAs) are nature inspired methods solving optimization problems, which employ computational models of evolutionary processes. Various evolutionary algorithms have been proposed in the literature. The most important ones are: Genetic Algorithms (Goldberg, 1989; Holland, 1975), Evolutionary Programming (Fogel, 1996; Fogel et al., 1966), Evolution Strategies (Hansen & Ostermeier, 2001; Rechenberg, 1994), Genetic Programming (Koza 1992), Particle Swarm Optimization (Kennedy & Eberhart, 1995) and Differential Evolution algorithms (Storn & Price, 1997). The algorithms mentioned above share the common conceptual base of simulating the evolution of a population of individuals using a predefined set of operators. Generally, the operators utilized belong to one of the following categories: the selection and the search operators. The most commonly used search operators are the mutation and the recombination.
In general, EA's are parallel and distributed implementations and they are inspired by niche formation. Niche formation is a common biological phenomenon (Baeck et al., 1997). Niches could aid the differentiation of the species by imposing reproduction restrictions. Many natural environments can lead to niche formation. For example, remote islands, high mountains and isolated valleys, restrict the species and therefore the evolution process. Although diversity tends to be low in each subpopulation, overall population diversity is maintained through isolation. However, occasionally an individual escapes and reaches nearby niches, further increasing the diversity of their populations (Baeck et al., 1997).
In this chapter, we study the class of Higher Order Neural Networks (HONNs) and in particular Pi-Sigma Networks (PSNs), which were introduced by Shin and Ghosh (1991a). Although, in general, PSNs employ fewer weights and processing units than HONNs, and manage to indirectly incorporate many of HONNs' capabilities and strengths. PSNs have effectively addressed several difficult tasks, where traditional Feedforward Neural Networks (FNNs) are having difficulties, such as zeroing polynomials (Huang et al., 2005) and polynomial factorization (Perantonis et al., 1998). Here, we study PSN's performance on several well known neural network training problems.
The novelty of the proposed approach is that the PSNs were trained with small integer weights and threshold activation functions, utilizing distributed Evolutionary Algorithms. More specifically, modified distributed versions of the Differential Evolution (DE) (Plagianakos & Vrahatis, 2002; Storn & Price 1997) and the Particle Swarm Optimization (PSO) (Clerck, 2006; Kennedy, & Eberhart, 1995) algorithms have been used. DE and PSO have proved to be effective and efficient optimization methods on numerous hard real-life problems (see for example Branke, 1995; Clerc, 2006; Jagadeesan et al., 2005; Magoulas et al., 2004; Parsopoulos & Vrahatis, 2004, 2002; Plagianakos et al., 2005, 1998; Plagianakos & Vrahatis, 1999, 2002; Storn, 1999; Tasoulis et al., 2004, 2005). The distributed EAs has been designed keeping in mind that the resulting integer weights and biases require less bits to be stored and the digital arithmetic operations between them are easier to be implemented in hardware. An additional advantage of the proposed approach is that no gradient information is required; thus (in contrast to classical methods) no backward passes were performed.