6.5 Summary

This chapter started by introducing the radial basis function (RBF) network as a two layer feedforward net employing hidden units with locally-tuned response characteristics. This network model is motivated by biological nervous systems as well as by early results on the statistical and approximation properties of radial basis functions. The most natural application of RBF networks is in approximating smooth, continuous multivariate functions of few variables.

RBF networks employ a computationally efficient training method that decouples learning at the hidden layer from that at the output layer. This method uses a simple clustering (competitive learning) algorithm to locate hidden unit receptive field centers. It also uses the LMS or delta rule to adjust the weights of the output units.

These networks have comparable prediction/approximation capabilities to those of backprop networks, but train by orders of magnitude faster. Another advantage of RBF nets is their lower "false-positive" classification error rates. However, by a factor of at least one order of magnitude, RBF nets require more training data and more hidden units compared to backprop nets for achieving the same level of accuracy.

Two major variations to the RBF network were given which lead to improved accuracy (though, at the cost of reduced training speed). The first variation involved replacing the k-means-clustering-based training method for locating hidden unit centers by a "soft competition" clustering method. The second variation to the RBF net substitutes semilocal activation units for the local activation hidden units, and employs gradient descent-based learning for adjusting all unit parameters.

The CMAC is another example of a localized receptive field net which was considered in this chapter. The CMAC was originally developed as a model of the cerebellum. This network model shares several of the features of the RBF net such as fast training and the need for a large number of localized receptive field hidden units for accurate approximation. It also has common features (and limitations) to those found in classical perceptrons (e.g., Rosenblatt's perceptron). One distinguishing feature of the CMAC, though, is its built-in capability of local generalization. The CMAC has been successfully applied in the control of robot manipulators.

Three unit-allocating adaptive multilayer feedforward networks were also described in this chapter. The first two networks belong to the class of hyperspherical classifiers. They employ hidden units with adaptive localized receptive fields. They also have sparse interconnections between the hidden units and the output units. These networks are easily capable of forming arbitrarily complex decision boundaries, with rapid training times. In fact, for one of these networks (PTC net), the training time was shown to be of polynomial complexity.

The third unit allocating network (cascade-correlation net) differs from all previous networks in its ability to build a deep net of cascaded units and in its ability to utilize more than one type of hidden units, co-existing in the same network. The motivation behind unit-allocating nets is two fold: (1) The elimination of the guesswork involved in determining the appropriate number of hidden units (network size) for a given task, and (2) training speed.

Finally, two examples of dynamic multilayer clustering networks are discussed: The ART1 net and the autoassociative clustering net. These networks are intended for tasks involving data clustering and prototype generation. The ART1 net is characterized by its on-line capability of clustering binary patterns, its stability, and its ability to follow nonstationary input distributions. Generalizations of this network allow the extension of these desirable characteristics to the clustering of analog patterns. These ART networks are biologically motivated and were developed as possible models of cognitive phenomena in humans and animals.

The second clustering net is motivated by "concept forming" cognitive models. It is based on two interrelated mechanisms: prototype formation and prototype extraction. A slightly modified backprop training method is employed in a customized autoassociative net of sigmoid units in an attempt to estimate and encode cluster prototypes in such a way that they become attractors of a dynamical system. This dynamical system is formed by taking the trained feedforward net and feeding its output back to its input. Results of simulations involving data clustering with these nets are given. In particular, results of motor unit potential (MUP) prototype extraction and noisy/distorted MUP categorization (clustering) for a real EMG signal are presented.

It is hoped that the different network models presented in this chapter, and the motivations for developing them give the reader an appreciation of the diversity and richness of these networks, and the way their development has been influenced by biological, cognitive, and/or statistical models.

Back to the Table of Contents

Back to Main Menu