Análisis del entorno

Before a network can be used, it needs to be trained. As mentioned previously, a network learns (e.g. is trained) by the systematic adjustment of the bias and weight values as determined by a learning algorithm (Haykin, 1994). While there are many types of learning suited to many types of neural networks, only two are of interest here. The one is supervised learning, by far the most popular type of learning, while the second is reinforcement learning, which is discussed in detail in Appendix B. A.2.1 Supervised Learning

The main ingredient of supervised learning is an external teacher, which provides the network with examples to learn from (Haykin, 1994). Whenever an input is pre- sented to the network, the teacher knows the target response. The difference between the output of the network and desired output (as provided by the teacher), called the error, is then used by the supervised learning algorithm to make appropriate adjustments to the network parameters. This is done incrementally, step-by-step, until some minimum error criterion is reached.

The most well-known supervised learning algorithm is the backpropagation al- gorithm, which is widely used to train networks for pattern recognition, system modelling, etc (Haykin, 1994). Learning can take place on-line or off-line. When learning takes place off-line, a separate computational facility is used to train the network until it meets the desired performance, and the design is then "frozen" for implementation (Haykin, 1994). For on-line learning, the network is adjusted while in use, and learning takes place in real time.

A drawback of supervised learning is that after learning, its behaviour is set to a certain pattern. Therefore, if new, unknown conditions are encountered, it cannot learn from these unless they are explicitly incorporated into the teacher and training is again undertaken. Reinforcement learning, discussed next, overcomes this limitation by not requiring an external teacher.

A.2.2 Reinforcement Learning

Reinforcement learning, as opposed to supervised learning, does not take place by example. Rather, it is learning by trail-and-error via interaction with the environ- ment. Learning is guided by a scalar performance index, or reinforcement signal. Appendix B further describes reinforcement learning.

Appendix B

Reinforcement Learning

Reinforcement learning is a form of artificial intelligence or machine learning, per- haps best defined as learning by interaction (Sutton & Barto, 1998). In essence, it is learning what to do in order to maximise a numerical reward signal (Sutton & Barto, 1998). The learner finds out what are the best actions in whichever situation by trying different ones; over time, the learner finds the actions which maximise the reward it receives. This can be contrasted with supervised learning, the most com- monly used form of machine learning, where the learner is presented with examples which it needs to copy, i.e. learning by example (Sutton & Barto, 1998). A typical example of such a situation is the training of neural networks for pattern recognition or data modelling. In reinforcement learning, the learner is not told what actions are best, it finds that out by trial and error.

B.1

Basic elements

The reinforcement learning framework consists of two main elements: the learning

agent and the environment (Sutton & Barto, 1998). As defined above, the learning

agent interacts with the environment, and in doing so learns more about it. In addition to the two main elements, four sub-elements can be identified: a policy, reward function, value function, and (optional) model of the environment (Sutton

& Barto, 1998).

The policy defines the behaviour of the agent at any time. It can be described as a mapping from the perceived environmental state to the actions to be taken in that state (Sutton & Barto, 1998). The policy is the core of the reinforcement learning agent, since it determines the agent’s behaviour. The policy can be a simple function, lookup table, or even a search process, and can be deterministic or stochastic (Sutton & Barto, 1998). A neural network can be used as policy, with the environmental state as input, and the agent’s actions as output. For this thesis, the policy is defined by a neural network. Since this is the case, the agent behaviour is determined by the specific values of the synaptic weights and biases.

The reward function essentially defines the goal of the reinforcement learning problem (Sutton & Barto, 1998). It returns a single number, called the reward, which indicates how desirable the current state is. The learning agent tries to maximise the total reward it receives in a given period of time. The reward the

agent receives can be compared to pleasure and pain in biological systems.

The value function specifies the expected accumulated reward an agent can re- ceive over time for starting in a given state (Sutton & Barto, 1998). Thus the value function represents the long-term desirability of a state. This can be contrasted with reward, which is immediate. Actions are chosen based on value judgments. Not all reinforcement learning methods require a value function. Some methods search directly in policy space using function optimization methods such as ge- netic algorithms, genetic programming or simulated annealing (Sutton & Barto, 1998). One of these methods is known as policy gradients, which is particularly well-suited to learning problems with high dimensions and continuous states and actions, such as chemical processes (Sehnke et al., 2009). Another highly success- ful genre of algorithms for performing policy search are Evolution Strategies (ES) algorithms (Heidrich-Meisner & Igel, 2009).