On Simulation Performance of Feedforward and NARX Networks Under Different Numerical Training Algorithms

Introduction

Artificial neural networks are adaptive nonlinear systems capable to approximate any function. Theoretically, a neural network can approximate a continuous function to an arbitrary accuracy on any compact set (Funahashi, 1989; Hornik, 1991; Cybenko, 1989). The backpropagation (BP) algorithm that was introduced by Rumelhart (1986) is the well-known method for training a multilayer feed-forward artificial neural networks. It adopts the gradient descent algorithm. In the basic BP algorithm the weights are adjusted in the steepest descent direction (negative of the gradient). However, the backpropagation neural network (BPNN) has a slow learning convergent velocity and may be trapped in local minima. In addition, the performance of the BPNN depends on the learning rate parameter and the complexity of the problem to be modelled. Indeed, the selection of the learning parameter affects the convergence of the BPNN and is usually determined by experience. Many faster algorithms were proposed to speed up the convergence of the BPNN. They fall into two main categories. The first category uses heuristic techniques developed from an analysis of the performance of the standard steepest descent algorithm. The second category uses standard numerical optimization techniques. The first category includes the gradient descent with adaptive learning rate, gradient descent with momentum, gradient descent with momentum and adaptive learning rate, and the resilient algorithm. In the standard steepest descent, the learning rate is fixed and its optimal value is always hard to find. The heuristic techniques allow the optimal learning rate to adaptively change during the training process as the algorithm moves across the performance surface. Therefore, the performance could be improved. The second category includes conjugate gradient, quasi-Newton, and Levenberg-Marquardt (LM) algorithm. In the conjugate gradient algorithms, a search is performed along conjugate directions; therefore the convergence is faster than steepest descent directions. Quasi-Netwon method often converges faster than conjugate gradient methods since it does not require calculation of second derivatives. For instance, it updates an approximate Hessian matrix at each iteration. Finally, The L-M method combines the best features of the Gauss-Newton technique and the steepest-descent method. It also converges faster than conjugate gradient methods since the Hessian Matrix is not computed but only approximated. For instance, it uses the Jacobian that requires less computation than the Hessian matrix.