A Pi-Sigma higher order neural network (Pi-Sigma HONN) is a type of higher order neural network, where, as its name implies, weighted sums of inputs are calculated first and then the sums are multiplied by each other to produce higher order terms that constitute the network outputs. This type of higher order neural networks have good function approximation capabilities. In this chapter, the structural feature of Pi-Sigma HONNs is discussed in contrast to other types of neural networks. The reason for their good function approximation capabilities is given based on pseudo-theoretical analysis together with empirical illustrations. Then, based on the analysis, an improved version of Pi-Sigma HONNs is proposed which has yet better function approximation capabilities.

Top## Background

There have been proposed a number of different kinds of higher order neural networks (HONNs). Most of them can be categorized into two groups based on how the higher order terms are produced in them. One group is the group of sigma-pi type higher order neural networks, and the other is the group of pi-sigma type higher order neural networks.

The first neural network named high-order neural network by Giles and Maxwell (1987) lies in the group of sigma-pi type HONNs. Each of its outputs is a weighted sum of higher order terms. In other words, higher order terms of the input signals are produced first (pi-operations), and then their weighted sums are calculated (sigma-operations). The group of sigma-pi type HONNs includes neural networks with Sigma Pi Units (Rumelhart, Hinton & McClelland, 1986) and neural networks with Product Units (Durbin & Rumelhart, 1989).

On the other hand, in a higher order neural network belonging to the pi-sigma group, the higher order terms are computed through sigma and pi operations but in reverse order: weighted sums of the input signals are calculated first, and then they are multiplied by each other. Pi-Sigma Networks and Ridge Polynomial Networks both proposed by Shin and Ghosh (1991; 1995) belong to this group. Moreover, if we focus on a distinctive feature of this group of HONNs which is not shared with the sigma-pi type HONNs, we can include the following networks in this group as well: Modular Networks (Jacobs & Jordan, 1993), Hierarchical Mixtures of Experts (HMEs) (Jordan, 1993), Neural Networks with Node Gates (Murata, Nakazono & Hirasawa, 1999; Myint, et al., 2000), Universal Learning Networks with Multiplication Neurons (Li, et al., 2001), Group Method of Data Handling (GMDH) (Farlow, 1984), and even Takagi-Sugeno type Fuzzy Models (Takagi & Hayashi, 1991).