PRUEBA DE HIPÓTESIS

In this thesis I have expounded the merits of a class of neural networks based on the use the Gaussian radial basis function. Where appropriate, comparisons were made with conventional networks from the same categories to assess the relative performance of the new networks. Without exception, the new class of networks performed significantly better than those one would normally use for a particular task using the tests given. However, it is impossible to test the networks comprehensively so that a claim can be made that the RBF networks will perform better in every circumstance and no attempt has been made to do this. The object of the simulations performed was to highlight in a simple way using patterns of obvious but not gratuitous complexity the attributes of the RBF structures. It is possible though to use the network on "real" data such as sampled speech to gain a better understanding of the networks capabilities in a practical environment and thus highlight any shortcomings that may become apparent. This is left as future work. As the training equations are incremental, the networks discussed will suffer to some degree from settling into a local minimum. Techniques could be explored to alleviate this by using information about the number of inputs "taken" by a particular node in relation to its region of influence (as both pieces of information are readily available). The local minimum problem is important and further comparisons are required with other networks and perhaps other techniques from related disciplines (eg. pattern recognition). To increase the convergence speed or accuracy, modifications to the training equations could be made. Also, work is required to assess what size of network is suitable for a particular task. This could be done by trying a number of candidate solutions and selecting the best or by increasing (or decreasing) the size of the network as training progresses and monitoring the improvement (if any !). The geometric figures used can also be modified, one example being an ellipsoid whose axes need not necessarily align with each dimension. One feature that has been touched upon during this thesis is the ability of the RBF networks described to be able to signal that it cannot comment on a particular input as the input is insufficiently close to the training data experienced or the number of nodes used is inadequate. It is my personal belief that this feature will become increasing important as the application areas for neural networks extends to domains that demand high integrity. Perhaps the most important thing to do now that a unified structure and training has been presented is to consider an appropriate implementation. An interesting possibility is the use of

controllable data paths, such that the configuration of the "processor" is optimum or near optimum in carrying out a specified series of operations concurrently. Central to this, is the notion that fixed point arithmetic is adequate for realising the neural network structures discussed. To verify this hypothesis, one could devise a comprehensive series of tests and run them using the neural network language translator (NNL). An outline sketch for the foundation of a pseudo-asynchronous re-configurable architecture (PARA) was discussed in chapter 7 and this requires more thought and detailed design before the practicality of the proposed architecture can be verified. This would make an interesting and potentially useful project within its own right. When this is completed (assuming the project is successful) then large networks can be cascaded and trials initiated on implementing the simple model of artificial perception referred to throughout this thesis. This again would make an excellent project Finally, alternative methods of constructing a neural network should be explored perhaps using analogue devices and perhaps other technologies (eg. optical).

From the above, it can be seen that the research documented throughout this thesis has generated more questions than answers. It is gratifying to note that some of the issues raised have spawned new research interests (see Grant 91, and Taylor 91) and I would like to take this opportunity to wish those involved every success in their endeavours.

[Aho86] [And83] [Barl90] [Ban-87] [Bat68] [Bat69] [Bat74] [Bat78] [Bou91] [Bro88] [Bro89] [Buf89] [Car86]

Aho, A.V., Sethi, R., Ullman, D., "Compilers principles, techniques and tools", Addison-Wesley world student series", 1986.

Anderson, J.A., "Cognitive and Psychological Computation with Neural models", IEEE Tran. System Man and Cybernetics, Vol SMC13, No5, pp799-815,1983

Barlow, H., "What does the brain see ? How does it understand", In Images and understanding, Ed's. H., Barlow, C., Blakemore, M., Weston- Smith, Cambridge University press, 1990.

Baron R.J., "The cerebral computer", Lawrence Erlbaum Associates, London, 1987.

Batchelor B.G, Wilkins, B. R."Adaptive discriminant functions", Pattern Recognition, I.E.E. Conf. Publication, 42, ppl68-178,1968.

Batchelor B.G, "Learning machines for pattern recognition", Ph.D Thesis, Southampton, 1969.

Batchelor, B. “Practical approach to pattern classification”, Plenum Press, 1974.

Batchelor B.G, "Classification and data analysis in vector spaces", in "Pattern Recognition ideas in Practice", Ed. Batchelor B.G, Plenum Press, 1978.

Bounds, D,, "Current neural network research", Linnet club presentation, University college London, 6th February, 1991.

Broomhead, D.S., Lowe, D., "Multivariable functional interpolation and adaptive networks", Complex systems, 2, pp321-355, 1988.

Broomhead, D.S., Jones, R., McWhirter, J., G., Shepherd, T., J., "A parallel architecture for non-linear adaptive filtering and pattern recognition", IEE Conference on Artificial neural networks, No 313, pp265-269,1989.

Buffa, M., "Nestor's decision learning system", The second European seminar on Neural computing , IBC technical services, London, England, February 1989.

Carpenter, G., Grossberg, S., “Neural dynamics of category learning and recognition etc”, Brain Structure, learning and memory (AAAS symposium), eds J. Davies et al.

[Car87] [Che86] [Che91] [Coh83] [Con86] [Den89] [Dud73] [Far85] [Faw86] [Gil87] [Graf86] [Graf8 8] [Gran86]

Carpenter, G., Grossberg, S., “A massively parallel architecture for a self-organizing pattern recognition machine”, Computer vision,Graphics and image processing, No 37, pp 54-115.

Chen, H., Lee, Y.C., Sun, G.Z., "High order model for associative memory", Proc. AIP conf. on neural networks for computing, Snowbird, pp86-99,1986.

Chen, S., Cowan, C., F., N., Grant, P., M., "Orthogonal least squares learning algorithm for radial basis function networks", IEEE Tran, on Neural Nets., Vol 2, No 2, March 1991.

Cohen, M., Grossberg, S., "Absolute stability of global pattern formation and parallel memory storage by competitive neural networks", IEEE Tran, on Systems, man and Cybernetics, Vol SMC-13, No 5, pp815-826,

1983.

Conrad, M., "The lure of molecular computing", IEEE Spectrum, Vol 23, No 10, pp55-60, 1986.

Denker J.S. "AT&T Neural Computing research", The second European seminar on Neural computing (commercial prospects), IBC technical services, London, England. February 1989.

Duda, R., Hart, P., "Pattern classification and scene analysis", J. Wiley & Sons, 1973.

Farhat, N. H., Psaltis, D., Prata, A., Paek, E., "Optical implementation of the Hopfield model", Applied Optics, Vol. 24, No. 10, ppl469-1475,

1985.

Fawcett, J., “Wired for thought”, New Scientist, 28 August 1986.

Giles, C.L., Maxwell, T.,"Learning invariance, and generalization in high-order neural networks" Applied Optics, Vol 26, No 23,1987.

Graf, H., Hubbard, W., Jackel, L., Howard, R., Straughn, B., Denker, J., Hubbard, W., Tennant, D., Schwartz, D., “VLSI implementation of a neural network memory with several hundreds of neurons", Proc. AIP conf. on neural networks for computing, Snowbird, pp86-99,1986.

Graf, H., Jackel, L., Hubbard, W., “VLSI implementation of a neural network model", IEEE Computer, Vol 21, March 1988.

Grant, P., M., Sage, P., "Acomparison of neural network and matched filter processing for detecting lines in images", Proc. AIP conf. on neural networks for computing, Snowbird, 1986.

[Gran91] Grant, D., Houselander, P., Taylor, J., T., "Realisation of binary associative memories using standardised current mode analogue building blocks", To be published in: IEEE Int. Symp. on Circ. & Syst., 1991. [Gro82] Grossberg, S., "Studies of mind and brain", Reidal, Boston MA, 1982. [Gro87] Grossberg, S., “Competitive learning: From interactive activation to

adaptive resonance”, Cognitive science, No 11, pp23-63.

[Han84] Hanks, P., "Collins dictionary of the English language", Editor P. Hanks, Collins, London, 1984.

[Har75] Hartigan, J., A., "Clustering Algorithms", J. Wiley & Sons, 1975.

[Heb49] Hebb, D., “Organization of behaviour”, New York science editions, 1949.

[Hec87a] Hecht-Neilsen R., "Kolmogorov's mapping neural network existance theorem", Proc. of the Int. Conf. on neural networks, vol 3, ppll-13, IEEE press, New York, 1987.

[Hec87b] Hecht-Neilsen R., "Counterpropagation networks", Applied Optics, vol 26, no 23,1987.

[Hec88] Hecht-Neilsen R., "Neurocomputing", IEEE Spectrum, pp36-41, March, 1988.

[Hec89] Hecht-Neilsen R., "Theory of the backpropagation network", Proc. of the Int. Conf. on neural networks, vol 1, pp593-611, IEEE press, New York, 1989.

[Hec90] Hecht-Neilsen R., "Neurocomputing", Addison-Wesley, 1990.

[Hop82] Hopfield J.J., “Neural networks and physical systems with emergent collective computational abilities”, Proc. Natl. Acad. Sci. 79, pp 2554- 2558, 1982.

[Hou85] Houselander, P., "The theory and approximation of recursive digital filters", Final year project, Dept. Elec. Eng., University of Essex, Wivenhoe park, Colchester, 1985.

[Hou88a] Houselander, P., Taylor, J., T., Haigh, D.,“A Current mode analogue circuit for implementing artificial neural networks”, IEE Elec. Letts, vol 24, 12 May 1988.

[Hou88b] Houselander, P.,Taylor, J., T., Haigh, D.,“An analogue Current mode circuit for implementing artificial neural networks”, IEE Saraga

colloquium, 1988.

[Hou88c] Houselander, P., Taylor, J., T., “Time-multiplexed analogue circuit for implementing artificial neural networks”, IEE Elec. Letts., Vol 24, No 23, pp 1413-1414,1988.

[Hou89a] Houselander, P., Taylor, J., T., “Current mode integrated circuit realisation of artificial neural networks”, IEE Colloquium on Current mode analogue circuits, 17 Feb. 1989, (Invited contribution).

[Hou89b] Houselander, P., Taylor, J., T., “On the use of pre-defined regions to minimise the training and complexity of multi-layer neural networks”, IEE Conference on Artificial neural networks, No 313, pp383-386,1989. [Hou90a] Houselander, P., Taylor, J., T., “Comparing the performance of the

Hebbian and Delta trained networks”, IEE Elec. Letts., Vol 26, No 2, pp 85-87,1990.

[Hou90b] Houselander, P., Taylor, J., T., “Improving the Hamming binary associative memory”, Electron. Lett, 26, pp 705-707,1990.

[Hou90c] Houselander, P., Taylor, J., T.,"Unifying the Hopfield & Hamming binary associative memories", IEEE Int. Symp. on Circ. & Syst., Vol 1, pp 188-191,1990.

[Hou90d] Houselander P., Taylor, J., T., "Calculating the Upper & Lower bounds on the capacity of two modified Hebbian trained Hopfield networks", Elec. Lett., Vol 26, No. 16, pp 1266 - 1267, Aug 1990.

[Hou91a] Houselander P., Taylor, J., T., "A self-organising Artificial neural network based on the RBF neuron", Submitted too, IEEE Tran. Neural networks., January 1991.

[Hou91b] Houselander P., Taylor, J., T., "Improving and characterising the performance of the Hamming binary associative memory", In preperation, 1991.

[Hua87] Huang, W., Lippmann, R., "Comparison between neural networks and conventional classifiers", IEEE 1st Conf. on Neural Networks, Vol 4, pp 485-493,1987.

[Hwa79] Hwang, K., "Computer arithmetic: Principles, Architecture, and design", Wiley, 1979.

[Jac87] Jackel, L., Graf, H., Howard, R., "Electronic neural network chips", Applied optics, Vol 26, No 23, pp5077-5080,1987.

[Joh88] Johnson-laird, P.N.,"The computer and the mind an introduction to cognitive science", Fontana masterguides, 1988.

[Kan82] Kandel, A., "Fuzzy techniques in pattern recognition", J. Wiley & Sons, 1982.

[Ker88] Kemighan, B., Ritche, D., "The C programming language, Second Edition" Prentice Hall software series, 1988.

[Kha90] Khana, T., "Foundations of neural networks", Addison-Wesley, 1990. [Kir83] Kirkpatrick, S., Gelatt, C., Vecchi, M., "Optimisation by simulated

annealing", Science, Vol.220, No4598,13 May 1983.

[Koh72] Kohonen, T., “Correlation matrix memories", IEEE Trans, on Comp., Vol c-21, No 4, pp353-359,1972.

[Koh88a] Kohonen, T., “Self-organization and associative memory”, Springer Verlag, Berlin 1988.

[Koh88b] Kohonen, T., “The neural phonetic typewriter”, IEEE Computer, Vol 21, March 1988.

[Kol57] Kolmogorov, A. N., "On the representation of continious functions of many variables by superposition of continious functions of one variable and addition", [in Russian], Dokl. Akad. Nauk USSR, 114, pp953-956, 1957.

[Kor88] Korst, J., Aarts, E., "Combinatorial Optimization on a Boltzmann Machine", The first European seminar on Neural computing , IBC technical services, London, England. February 1988.

[Kos87] Kosko, B., "Adaptive bidirectional associative memories", Applied Optics, Vol 26, No23, pp4947-4959, 1987.

[Kos88] Kosko, B., "Bidirectional associative memories", IEEE Tran, on SMC, Vol 18, Nol, pp49-60,1988.

[Lin86a} Linsker, R., "From basic network principles to neural architecture: Emergence of Spatial-opponent cells", Proc. Natl. Acad. Sci., Vol 83, pp7508-7512,1986.

[Lin86b} Linsker, R., "From basic network principles to neural architecture: Emergence of orientation-selective cells", Proc. Natl. Acad. Sci., Vol 83, pp8390-8394,1986.

[Lin88] [Lip87a] [Lip87b] [Lit78] [Low89] [Luc89] [Mac87] [Mak85] [Mas84] [McC43] [McE87] [Mea80] [Mea89]

Emergence of orientation columns", Proc. Natl. Acad. Sci., Vol 83, pp8779-8783,1986.

Linsker, R., "Self-organization in a perceptual network", IEEE Computer, pp 105-117, March 1988.

Lippmann R., “An introduction to computing with Neural Nets”, IEEE ASSP Magazine, April 1987.

Lippmann R., Gold, B., Malpass, M., “A comparison of Hamming and Hopfield neural networks for pattern classification", MIT Lincoln laboratory, Technical report 769,21st May 1987.

Little, W., Shaw, G., "Analytic study of memory storage capacity of a neural network", Mathematical Biosciences, Vol 39, pp281-290,1978. Lowe, D., "Adaptive Rasial Basis function nonlinearities, and the problem of generalisation", IEE Conference on Artificial neural networks, No 313, ppl71-175, 1989.

Lucas, A., Kittler, J., "A comparitive study of the Kohonen and multiedit neural net learning algorithms", IEE Conference on Artificial neural networks, No 313, pp l7 1-175, 1989.

Macukow, B., Arsenault, H., "Optical associative memory model based on neural networks having variable interconnection weights", Applied Optics, Vol. 26, No. 5, pp924-928,1987.

Makoul, J., Roucos, S., Gish, H.,"Vector quantization in speech coding", Proc. IEEE, Nov., ppl551-1588, 1985.

Massara, R., "Numerical design methods", Lecture notes for undegradutae course, Dept. Elec. Eng. University of Essex, 1984.

McCulloch, W., Pitts, W., “A logical calculus of the ideas imminent in nervous activity”, Bulletin of mathematical biophysics, No 5, pp ll5 - 133,1943.

McEliece, E., Posner, E., Rodermich, E., Venkatesh, S.,“The capacity of the Hopfield associative memory”, IEEE Tran, on information theory, Vol IT-33, No 4, pp461-482,1987.

Mead, C., Conway, L., "Introduction to VLSI systems", Addison- Wesley, 1980.

[Mic86] Micchelli, C. A., "Interpolation of scattered data: Distance matrices and conditionally positive definite functions", Constructive Approximation, 2, pp 11-22, 1986.

[Mic89] Michalski, G., "The U.S. Neural-Network Market", The second European seminar on Neural computing , IBC technical services, London, England. February 1989.

[Min69] Minsky M., Papert S., “Perceptrons: An introduction to Computational Geometry”, MIT Press, 1969.

[Mur89] Murray, A., F., "Silicon implementations of neural networks”, IEE Conference on Artificial neural networks, No 313, pp27-32,1989

[Nil66] Nilsson, N., “Learning machines”, McGraw Hill, N.Y., 1965.

[Nob86] Noback, C., Demarest, R.,"The nervous system introduction and review", 3rd edition, McGraw Hill, 1986.

[Oh88] Oh S., Yoon T., Kim J.C., “Associative-memory model based on neural networks: modification of Hopfield model”, Optics letters, Vol 23, No 1, pp 74-76,1988.

[Par82] Parker, D., “Learning logic”, Invention report S81-64, File 1, Office of Technology Licensing, Stanford University, 1982.

[Pen89] Penrose, R., "The Emperor's new mind", Vintage, 1989.

[Pog84] Poggio, T.,"Vision by man and machine",Scientific American, April, 1984.

[Pow85] Powell, "Radial basis functions for multivariable interpolation: A review", IMA conference on "Algorithms for the approximation of functions and data", RMCS Shrivenham, 1985.

[Prad88] Prados D.L.,"Capacity of a neural network", IEE Elec. Letts., Vol 24, No8, pp454-455, 1988.

[Prad89] Prados D.L., Kak S.C. “Neural Network capacity using Delta Rule”, IEE Elec. Letts., Vol 25, pp 197-199,1989.

[Prag89] Prager R.W., Fallside F. (1989), “The modified Kanerva model for automatic speech recognition”, Computer speech and language, Vol 3, pp 61-81,1989.

[Rab75] Rabiner, L R., Gold, B., "Theory and application of digital signal processing", Prentice-Hall, 1975.

[Red83] Reddy, R., Zue, V., “Recognizing continuous speech remains an elusive goal” IEEE Spectrum, pp84-87, November 1983.

[Ric87] Richfield, S., "A logarithmic vector processor for neural nets", IEEE 1st Conf. on neural networks, Vol 4, pp 733-740,1987.

[Ros58] Rosenblatt, F., "A probabilistic model for information storage and organization in the Brain11, Psychological Rev., Vol 65, No 6, pp386-408,

1958.

[Ros62] Rosenblatt, F., “Principles of Neurodynamics: Perceptrons and the theory of Brain mechanisms” New York, Spartan Books, 1962.

[Rum86] Rumelhart, D., McClelland J., (Eds.), “Parallel Distributed Processing: Explorations in the microstructure of cognition”, MIT Press, 1986.

[Sag86] Sage, J., Thompson, K., and Withers, R., “An artificial Neural network integrated circuit based on MNOS/CCD principles”, Proc. AIP conf. on neural networks for computing, Snowbird, pp381-385,1986.

[Sam77] Sampath, G., Srinivasan, S. K.,"Lecture notes in biomathematics", Springer-verlag, 1977.

[Sel89] Selviah, D.R., Midwinter, J.E., "Correlating matched filter model for the analysis and design of neural networks", IEE Proceedings, 136, ppl43- 148,1989.

[Sej86] Sejnowski, T.J., "Higher order Boltzmann machines", Proc. AIP conf. on neural networks for computing, Snowbird, pp398-403,1986.

[Siv85] Sivilotti, M., Emerling, M., R., Mead, C., "A novel associative memory implemented using collective computation", Chapel Hill conf. on VLSI, pp329-342,1985.

[Stu58] Stuart, A., "Equally correlated variates and the Multinormal integral", J. R. Stat. Soc. Series B, No 20, pp373-378,1958.

[Tai88] Tai, H., Jong, T., "Neural networks with higher-order nonlinearity", IEE Elec. Letts., Vol 24, No 19, ppl225-1226,1988.

[Tat89] Tattersall, G., D., "Neural map applications", in Neural computing architectures, Ed. I., Aleksander, North Oxford Academic Publishers Ltd, pp41-73,1989.

[Tay89] Taylor, J. G., "A model of early vertebrate visual processing", IEE Conference on Artificial neural networks, No 313, ppl 15-119,1989.

[Tay91] [Trei84] [Trel88] [Trel89] [Tur36] [Von58] [Was89] [Way89] [Wea4 6] [Wer74] [Wid60] [Wid62] [Wid88] [Yu90]

Taylor, J., T., Grant, D., Houselander, P., "Synthesis of a class of of artificial neural network using a CMOS current mode building block approach", To be published in: IEE Colloq. Advances in analogue VLSI, 1991.

Treisman, A.,"Features and objects in visual processing",Scientific American, April, 1984.

Treleaven, P., "Parallel architectures for neurocomputers", The first European seminar on Neural computing , IBC technical services, London, England. February 1988.

Treleaven, P., Pacheco, M., Vellasco, M., "VLSI architectures for neural networks", IEEE Micro, Vol 9, No 6, pp8-27, December 1989.

Turing, A., "On computable numbers with an application to the Entscheidungsproblem", Proc. of Lond. Math. Soc., 2nd series, 42, pp230-265, 1936.

Von Neumann, J., “The computer and the brain”, Yale university press, 1958.

Wasserman P., “Neural computing theory and practice”,Van Nostrand Reinhold, 1989.

Wayner, P., "VLIW heir to RISC", BYTE magazine, Aug. 1989.

Weatherbum, C.,"A first course in mathematical statistics", Camb. Univ. Press, 1946.

Werbos, P., “Beyond regression: New tools for prediction and analysis in the behavioural sciences”, Dept. Appl. Math., Harvard University, 1974. Widrow, B., Hoff, M., "Adaptive switching circuits" IRE WESCON convention record, pp96-104, New York, 1960.

Widrow, B., "Generalisation and information storage in networks of AD ALINE neurons", in Self-Organizing systems, Spartan books, Washington DC, 1962.

Widrow, B., "Neural nets for adaptive filtering and adaptive pattern recognition", IEEE Computer, Vol 21, March 1988.

Yu, X., Cheng, S., "Training algorithms for backpropagation neural networks with optimal descent factor", IEE Elec. Letts., Vol 26, No 20, ppl698-1700, 1990.

This user guide details the grammar and function of the keywords used in the neural network language (NNL). The NNL program is translated into 'c' source code which conforms strictly to the ANSI specification and is therefore portable. The NNL translator is written in BBC Basic 5 and is designed for use in the RISCOS environment on the Archimedes manufactured by Acorn computers. The translator is therefore non-portable. The reason for using Basic 5 is that this implementation of Basic contains powerful constructs for string manipulation which has minimised the programming effort required. The translator is not listed here as the actual code is unimportant. What is of importance is the NNL, its syntax and the corresponding 'c' program produced and these we detail here.