Dynamics of Neural Networks as Nonlinear Systems with Several Equilibria

Introduction

Neural networks are computing devices for Artificial Intelligence (AI) belonging to the class of learning machines (with the special mention that learning is viewed at the sub-symbolic level). The basic feature of the neural networks is the interconnection of some simple computing elements in a very dense network and this gives the so-called collective emergent computing capabilities. The simple computing element is here the neuron or, more precisely, the artificial neuron – a simplified model of the biological neuron.

Artificial Neural Networks structures are broadly classified into two main classes: recurrent and non-recurrent networks. We shall focus on the class of recurrent neural networks (RNN). Due to the cyclic interconnections between the neurons, RNNs are dynamical nonlinear systems displaying some very rich temporal and spatial qualitative behaviors: stable and unstable fixed points, periodic, almost periodic or chaotic behaviors. This fact makes RNN applicable for modeling some cognitive functions such as associative memories, unsupervised learning, self-organizing maps and temporal reasoning.

The mathematical models of neural networks arise both from the modeling of some behaviors of biological structures or from the necessity of Artificial Intelligence to consider some structures which solve certain tasks. None of these two cases has as primary aim stability aspects and a “good” qualitative behavior. On the other hand, these properties are necessary and therefore important for the network to achieve its functional purpose that may be defined as “global pattern formation”. It thus follows that any AI device, in particular a neural network, has to be checked a posteriori (i.e. after the functional design) for its properties as a dynamical system, and this analysis is performed on its mathematical model.

A common feature of various RNN (automatic classifiers, associative memories, cellular neural networks) is that they are all nonlinear dynamical systems with multiple equilibria. In fact it is exactly the equilibria multiplicity that gives to all AI devices their computational and learning capabilities. As pointed out in various reference books, satisfactory operation of a neural network (as well as of other AI devices) requires its evolution towards those equilibria that are significant in the application.

Let us remark here that if a system has several isolated equilibria this does not mean that all these equilibria are stable – they may be also unstable. This fact leads to the necessity of a qualitative analysis of the system’s properties. Since there are important both the local stability of each equilibrium point and also (or more) the global behavior of the entire network, we shall discuss here RNN within the frameworks of the Stability Theory and the Theory of Systems with Several Equilibria.

The chapter is organized as follow. The Background section starts with a presentation of RNN from the point of view of those dynamic properties (specific to the systems with several equilibria), which make them desirable for modeling the associative memories. Next, there are provided the definitions and the basic results of the Theory of Systems with Several Equilibria, discussing these tools related to the Artificial Intelligence domain requirements. The main section consists of two parts. In the first part it is presented the basic tool for analyzing the desired qualitative properties for RNN as systems with multiple equilibria. The second part deals with the effect of time-delays on the dynamics of RNN. Moreover, one will consider here the time-delay RNN under forcing stimuli that have to be “reproduced”(synchronization). The chapter ends with Conclusions and comments on Future trends and Future research directions. Additional reading is finally suggested.