= M (1.11)
This is a special case of the Hebbian learning with the output signal being replaced by the desired signal But the Hebbian learning is an unsupervised learning, whereas the correlation learning is a supervised learning, since it uses the desired output value to adjust the weights. In the implementation of the learning law, the weights are initialised to small random values close to zero, 0.
1.6.6 (Winner-take-all) Learning Law
This is relevant for a collection of neurons, organized in a layer as shown in Figure 1.8. All the inputs are connected to each of the units
Figure 1.8 Arrangement of units for learning', where the adjusted weights are highlighted.
in the output layer in a manner. For a given input vector a, the output from each unit i is computed using the weighted sum
The unit k that gives maximum output is identified. That is
T a = max T a) (1.12)
i
Then the weight vector leading to the kth unit is adjusted as follows:
=
(a
- (1.13)Therefore,
( a for = M (1.14)
The final weight vector tends to represent a group of input vectors within a small neighbourhood. This is a case of unsupervised learning. In implementation, the values of the weight vectors are initialized to random values prior to learning, and the vector lengths are normalized during learning.
1.6.7 Learning Law
The learning law is also related to a group of units arranged in a layer as shown in Figure 1.9. In this law the weights are adjusted so as to capture the desired output pattern characteristics. The adjustment of the weights is given by
Basic Learning Laws
Figure 1.9 Arrangement of units for learning', where the adjusted weights are highlighted.
where the unit is the only active unit in the input layer. The vector b = is the desired response from the layer of
M units. The learning is a supervised learning law, and it is used with a network of to capture the characteristics of the input and output patterns for data compression. In ,implementation, the weight vectors are initialized to zero prior t o
1.6.8 Discussion on Basic Learning Laws
Table 1.2 gives a summary of the basic learning laws described so Table 1.2 Summary of Basic Learning Laws (Adapted from [Zurada, 19921)
Learning Weight adjustment Initial Learning
law weights
T
Hebbian = a) Near zero Unsupervised
=
for = 1, 2,
= - Random Supervised
= -
for = M
Delta = - Random Supervised
=
for j = M
- a,, Random Supervised
Hoff for = 1, 2,
Correlation = Near zero Supervised
for = 1, 2, M
Winner- Random but Unsupervised
take-all k is the winning unit, for j = 1, 2, M
= (b, - Zero Supervised
Basics of Artificial Neural Networks far. I t shows the type of learning and the nature of the output function for discrete1 for continuous) for which each law is applicable. The most important issue in the application of these laws is the convergence of the weights to some final limit values as desired. The convergence and the limit values of the weights depend on the initial setting of the weights prior to learning, and on the learning rate parameter.
The Hebb's law and the correlation law lead to the sum of the correlations between input and output (for Hebb's law) components and between input and desired output (for correlation law) components, But in order to achieve this, the starting initial weight values should be small random values near zero. The learning rate parameter should be close to one. Typically, the set of patterns are applied only once in the training process. In some variations of these (as in the principal component learning to be discussed in Chapter the learning rate parameter is set to a small value 1) and the training patterns are applied several times to achieve convergence.
The perceptron, delta and LMS learning laws lead to steady state values (provided they converge), only when the weight adjustments are small. Since the correction depends on the error between the desired output and the actual output, only a small portion of the error is used for adjustment of the weights each time. Thus the learning rate parameter 1. The initial weights could be set to random values. The set of training patterns need to be applied several times to achieve convergence, if it exists. The convergence will naturally be faster if the starting weights are close to the final steady values.
The weights and learning laws converge to the mean values of a set of input and desired output patterns, respectively. In these cases the learning rate parameter is typically set to a value less than one 1). The weights in the case of can be initialized to any random values, and in the case of
to small random values near zero. The set of training patterns are applied several times to achieve convergence.
Besides these basic learning laws there are many other learning laws evolved primarily for application in different situations 1995, Ch. 31. Some of them will be discussed at appropriate places in the later chapters.
1.7
Summary
In this chapter we have seen the motivation and background for the current interest in the study of problems based on models using artificial neural networks. We have reviewed the features of the biological neural network and discussed the feasibility of realizing
Review Questions 37 some of these features through parallel and distributed processing models (Appendix A). In particular, the associative memory, fault tolerance and concept learning features could be demonstrated through these PDP models. Some key developments in artificial neural networks were presented to show how the field has evolved to the present state of understanding.
An artificial neural network is built using a few basic building blocks. The building blocks were introduced starting with the models of artificial neurons and the topology of a few basic structures. While developing artificial neural networks for specific applications, the weights are adjusted in a systematic manner using learning laws. We have discussed some basic learning laws and their characteristics. But the full potential of a neural network can be exploited if we can incorporate in its operation the neuronal activation and synaptic dynamics of a biological neural network. Some features of these dynamics are discussed in the next chapter.
Review Questions
1. Describe some attractive features of the biological neural