Una breve descripción etnográfica del äp’e, äp’e

The backpropagation algorithm learns the connection weights for a multi-layer perceptron given a network with a fixed set of units and interconnections. In other words, the architecture of the neural network is known in advance. To use a backpropagation algorithm for training, the MLP must have at least one hidden layer, but not necessarily be fully connected. Table 2.11 presents a backpropagation algorithm for learning a three-layer feedforward MLP network using the sigmoid function as an activation function, adapted from Mitchell (1997).

BACKPROPAGATION Algorithm (training_examples, η, n_in, n_out, n_hidden)

{Each training example is a pair of the form xr,tr , where xr is the vector of network input values, and tr is the vector of target network output values. η is the learning rate. is the number of network inputs. is the number of output units. n is the number of units in the hidden layer. The input from unit i into unit j is denoted . The weight from unit i into unit j is denoted as .}

in n n_out w hidden ji x ji

Create a feed-forward network with n_in inputs, n_hidden hidden units, and n_out outputs. Initialise all connection weights to small random numbers (e.g., between -.05 and .05). Until the termination condition is met, Do

For each xr,tr in training_examples, Do

Propagate the input forward through the network:

1. Input the instance xr to the network and compute the output o of every unit in the network.

u u

Propagate the errors backward through the network and update weights:

2. For each network output unit , calculate its error term k δ_k

δ_k ←o_k(1−o_k)(t_k −o_k) 3. For each hidden unit , calculate its error term h δ_h

∑

4. Update each network weight w_ji

w_ji ← w_ji + ∆w_ji

where

∆w_ji =ηδ_jx_ji

TABLE 2.11 Backpropagation algorithm for multi-layer perceptron learning. The trained MLP

network is a three-layer feedforward neural network as shown in Figure 2.6, with a sigmoid function used as activation functions for processing units.

As shown in table 2.11, the backpropagation algorithm updates the weights of output layer units using the error between the target output and the actual output from the network as shown in step 2. This error is then propagated backwards to a hidden layer unit in the form of a weighted sum of all connections between this hidden layer unit and all output layer units. Backpropagation is essentially a gradient descent method that minimises the error between the target output and actual output from the network. For mathematical analysis of the backpropagation algorithm, please refer to Fausett (1994) and Russell and Norvig (2003, pp. 744-748).

Besides multi-layer perceptrons with backpropagation learning, many other neural network topologies have also been explored by different researchers. Recurrent networks are neural networks that possess at least one feedback connection. This is different from feedforward networks where signals only flow through the network in a forward direction. Some signals in a recurrent network flows in a backward direction. To illustrate this concept, Figure 2.7 shows a Simple Recurrent Network (SPN) (Elman 1990). The simple recurrent network is largely similar to a three-layer feedforward multi-layer perceptron except that there is a set of “context units” in the input layer. However, there are also connections from the hidden layer back to the context units fixed with a weight of 1 as shown as curved lines in Figure 2.7.

FIGURE 2.7 A simple recurrent neural network.

Input Units Hidden Units

Output Units

Context Units

At each time step, the inputs from training data are propagated in a standard feedforward fashion, and then a learning rule (usually backpropagation) is applied. The fixed feedback connections between the hidden units and context units result in the context units always maintaining a copy of the previous values of the hidden units (since they propagate over the connections before the learning rule is applied). In other words, these context units keep a memory of information in previous time steps. This makes particular sense when the network is trained to predict a time series. For example, when a recurrent neural network is used to predict stock prices, the feedback connections will enable the network to look back further in the price history. Recurrent networks are powerful for learning complex sequences, but obviously, recurrent networks have more complex network structures, and hence, can be difficult to design and train. Several training methods have been proposed for learning recurrent networks, such as the gradient-based method proposed in Williams and Zipser (1995). Hopfield networks (Hopfield 1982) are recurrent neural networks in which all connections are symmetric, i.e., if there is a connection from unit i to unit j, there must be a connection from unit j feedback to unit i. Other popular neural network models include Radial basis function (RBF) networks (Park and Sandberg 1991), probabilistic neural networks (PNN) (Specht 1990), and Kohonen self- organizing maps (SOM) (Kohonen 1997). Good introductions to neural netowrks can be found in Anderson and Rosenfield (1988), Fausett (1994) and Callan (1999).

Multi-layer perceptrons with backpropagation learning have proven successful in many practical problems such as financial time series predictions (Zirilli 1996), computer game playing (Tesauro and Sejnowski 1989), and industrial applications

(Schlang et al. 1996). However, backpropagation training on multi-layer perceptrons has a few drawbacks. First, backpropagation learning is a gradient descent method that minimises the mean square error between target and actual outputs over all trainning examples. It tends to become trapped in a local minimum and is incapable of finding global minima if its error function is multimodal and/or indifferentiable (Sutton 1986). Secondly, using backpropagation for neural network training requires the network archituecure to be known in advance. There are no generally accepted protocols in ANN research that specifies how many layers and hidden units and what activation functions should be used for a particular application. Developers usually design a network either based on their experiences or through tedious trials on different number of layers and hidden units. With attempts to solve these problems, evolutioanry approaches for learning artificial neural networks have been explored.

2.4.3 Evolutionary artificial neural networks

Yao (1999) describes an evolutionary learning approach that can be introduced into ANN’s at three different levels: connection weights, network architectures, and

learning rules. Using evolutionary algorithms, such as genetic algorithms, to evolve

connection weights provides a global search method for network weight training, and avoids the problem of being trapped in a local minimum caused by gradient descent learning. Evolving network architectures without human intervention provides an approach to automatic ANN design as both ANN connection weights and structures can be evolved. The evolution of learning rules can regarded as a process of “learning how to learn” in ANN’s where the adaptation of learning rules is achieved through