Introduction

In this project, we introduce a novel learning rule by applying Rousseeuw's least trimmed squares (LTS) method to increase the robustness of feedforward neural networks. The particular paradigm of networks under consideration is the radial basis function network (RBFN); it has significant advantages over multilayer perceptrons (MLP), namely faster convergence, smaller extrapolation errors, higher reliability, and a more well-developed theoretical analysis.
Similar robust learning rules that have been applied to different paradigms, for example, MLP and GMDP networks. In order to compensate for the loss of efficiency, an adaptive version of the proposed learning rule is developed in an attempt to maximize the robustness and efficiency. In contrast to most present robust learning rules, our method neither is dependent upon any assumptions about error distribution nor is it necessary to estimate the error distribution. More importantly, an upper bound on the number of outliers that the resulting robust RBFN can withstand is established.

Robust Radial Basis Function Networks

Radial basis function networks (RBFNs) have increasingly attracted interest for engineering applications due to their advantages over traditional multilayer perceptrons, namely faster convergence, smaller extrapolation errors, and higher reliability. RBFN is a class of single hidden layer feedforward networks where the activation functions for hidden units are defined as radially symmetric basis functions phi such as the Gaussian function.
Each hidden unit forms a localized receptive field in the input space X, whose centroid is located on c with width sigma. The fraction of overlap between each hidden unit and its neighbors is decided by the width sigma such that a smooth interpolation over the input space is allowed. Therefore, unit i gives a maximum response for input stimuli close to c. In general, training RBFNs involves two phases: one is unsupervised learning (clustering) on the hidden layer to determine N receptive field centroids in the training data set and the associated widths; the other is supervised learning on the output layer to estimate the connection weights w. The output layer simply implements a multiple linear regression and hence can be trained by the iterative gradient descent method based on the least squares approach. Much work has been devoted to improving the learning performance for clustering and determining the optimal complexity for networks since it plays a critical role in achieving good approximation precision although the choice of the radial basis functions is not crucial for performance; for example, see (CHENS-91, MOOD-89). In contrast to this work, we focus on the issue of enhancing reliability of RBFNs in the presence of gross errors.

Experimental Results and Analysis

A number of simulations have been conducted to evaluate the efficiency and the stability against the presence of outliers for the proposed AR2BF learning rule. Simulation results using the aforementioned learning rules, namely RBF, R2BF, and AR2BF rules on the problems of 1-D and 2-D function approximation will be discussed.

Bibliography

  • [BURR-91] P. Burrascano, "A Norm Selection Criterion for the Generalized Delta Rule", IEEE Trans. on Neural Networks, vol. 2, pp. 125-130, 1991.

  • [BURR-94] P. Burrascano, "Robust Learning in the presence of Outliers", 1994 International Symposium on Artificial Neural Networks, Taiwan, 1994.

  • [CHEN-94] D. S. Chen and R. C. Jain, "A Robust Back Propagation Learning Algorithm for Function Approximation", IEEE Trans. on Neural Networks, vol. 5, pp. 467-479, 1994.

  • [CHENS-91] Chen, S, Cowan, C. F. N., and Grant, P. M., "Orthogonal Least Squares Learning Algorithm for Radial Basis Function Networks", IEEE Trans. on Neural Networks, vol. 2, No. 2, pp. 302-309, 1991.

  • [CICH-93] A. Cichocki and R. Unbehauen, "Robust Estimation of Principal Components by using Neural Network Learning Algorithms", IEE Electronics Letters, vol. 29, pp 1869-1870.

  • [COVE-65] T. M. Cover, "Geometrical and Statistical Properties pf Systems of Linear Inequalities with Applications in Pattern Recognition", IEEE Trans. on Electronic Computers, vol. EC-14, pp. 326-334, 1965.

  • [GEMA-92] S. Geman and E. Bienenstock, "Neural Networks and the Bias/Variance Dilemma", Neural Computation, vol. 4, pp. 1-58, 1992.

  • [GIRO-92] Federico Girosi, "Some Extensions of Radial Basis Functions and Their Applications in Artificial Intelligence", Computers Math. Applic., vol. 24, no. 61-80, 1992.

  • [HAMP-85] F. R. Hampel, E. M. Ronchetti, P. J. Rousseeuw, and W. A. Stahel, Robust Statistics: The Approach Based on Influence Functions, John Wiley & Sons, Inc., 1985.

  • [HART-90] E. J. Hartman, J. D. Keeler and J. M. Kowalski, "Layered Neural Networks with Gaussian Hidden Units as Universal Approximators", Neural Computation, vol. 2, pp. 210-215, 1990.

  • [HORN-89] K. Hornik, M. Stinchcombe, and H. White, "Multilayer Feedforward Networks are Universal Approximators", Neural Networks, vol. 2, pp. 359-366, 1989.

  • [HWAN-92] J. N. Hwang, H. Li, M. Maechler, D. Martin, and J. Schimert, "Projection Pursuit Learning Networks for Regression", Engr. Applic. Artificial Intelligence, vol. 5, pp.193-204, 1992.

  • [JOIN-92] J. A. Joines and Mark W. White, "Improved Generalization Using Robust Cost Functions", Proc. IEEE ICNN, vol. 3, pp. 911-918, 1992.

  • [KARA-92] N. B. Karayiannis and A. N. Venetsanopoulos, "Fast Learning Algorithms for Neural Networks", IEEE Trans. on Circuits and Systems, vol. 39, pp. 453-474, 1992.

  • [KIM-94] J. Kim and J. H. Seo, "Implementation of a Robust Feedforward Neural Network Using the Classifier Structure", IEEE ICNN, vol. 3, pp. 1427-1432.

  • [LI-94-0] S. Li, Y. Chen, and E. L. Leiss, "GMDP: a Novel Unified Neuron Model for Multilayer Feedforward Neural Networks", Proc. IEEE ICNN, Orlando, pp. 107-112, 1994.

  • [LI-94-1] S. Li and E. L. Leiss, "Robust Learning in Feedforward Neural Networks with GMDP Units", 1994 International Symposium on Artificial Neural Networks, Taiwan, 1994.

  • [LI-94-2] S. Li and E. L. Leiss, "Enhancing Robust Learning in Multilayer Perceptrons", The Sixth International Conference on Artificial Intelligence and Expert Systems Applications, Houston, 1994.

  • [LIAN-94] K. Liano, "A Robust Approach to Supervised Learning in Neural Network", Proc. IEEE ICNN, Orlando, vol. 1, pp. 513-516, 1994.

  • [MEER-91] P. Meer, D. Mintz, A. Rosenfeld, and D. Y. Kim, "Robust Regression Methods for Computer Vision: A Review", Internation Journal of Computer Vision, vol. 6, pp. 59-70, 1991.

  • [MOOD-89] J. E. Moody and C. J. Darken, "Fast Learning in Networks of Locally-tuned Processing Units", Neural Computation, vol. 1, pp. 281-294, 1989.

  • [NILS-90] N. J. Nilsson, The Mathematical Foundations of Learning Machines, Morgan Kaufmann Publishers, 1990.

  • [PARK-91] J. Park and I. W. Sandberg, "Universal Approximation using Radial-Basis Function Networks", Neural Computation, vol. 3, pp. 246-257, 1991.

  • [PARL-94] A. G. Parlos, B. Fernandez, A. F. Atiya, J. Muthusami, and W. K. Tsai, "An Accelerated Learning Algorithm for Multilayer Perceptron Networks", IEEE Transactions on Neural Networks, vol. 5, pp. 493-497, 1994.

  • [POGG-90] Poggio, T. and Girosi, F., "Networks for Approximation and Learning", Proc. of the IEEE, vol. 78, No. 9, pp. 1481-1497, 1990.

  • [ROUS-84] P. J. Rousseeuw, "Least Median of Squares Regression", Journal of the American Statistical Association, vol. 79, pp. 871-880, 1984.

  • [ROUS-87] P. J. Rousseeuw and A. M. Leroy, Robust Regression and Outlier Detection, John Wiley \& Sons, Inc., 1987.

  • [WHIT-89] White, H., "Learning in Artificial Neural Networks: A Statistical Perspective", Neural Computation, vol. 1, pp. 425-464, 1989.