with the previous two requirements for ease of discrimination and
generalisation.
Chapter 7 Training Heuristics and their Implementation 7.6
These requirements may lead to. the use of fuzzy targets, input coding and pre processing. Consideration of the S-R characteristics is part of the task analysis stage in instructional design(for people) to determine whether a specialised treatment will be required. A similar approach is possible in instructional design for artificial neural networks. In ID for ANNs consideration of S-R characteristics may allow a choice of which inputs and outputs to use, how to represent them and what coding scheme to use. The extent to which these are under control of the neural network instructional designer will vary, however at least some of these aspects are generally not fixed by the nature of the problem. By carefull selection of which inputs and outputs are used and how they are represented (or coded) the network performance may be improved. A network which would not previously converge may then do so, convergence may be faster or the solution accuracy may be increased. Where certain S-R characteristics cannot be avoided, their presence may suggest the need for specialised training regimes which make use of the heuristics covered in the following sections of this chapter. Finally, it may be possible to pre-process the inputs before presenting them to the network. This is done to improve the S-R characteristics of the task as to the network. Self organising networks have been shown to provide useful transformations for this pre-processing stage [Hrycej 1 992, Kohonen 1990, Grandvalet et al. 1991] but analytical and algorithmic methods can also be used.
The implementation of each of these techniques is considered in the following p aragraphs.
7 .2 . 1 . 1
The application of an ANN to any specific task involves the choice of input output coding techniques. These choices should be made explicit while keeping the discrimination and generalisation requirements in mind. That is the choice of coding techniques should be taken as a specific step in applying a network to each new task. The choice should be made so as to ensure that
a) the input vectors for instances which are to provide similar responses
are as similar as possible.
b) the input vectors for instances which are to provide different
Chapter 7 Training Heuristics and their Implementation 7.7
This can be achieved by choosing from the following forms of coding;
• Direct
- A single number is used for each network input or output,
normalised to the range of either 0-1 or -1 to + 1 depending on the
squashing function used in the neuron model. For example in the
NNWF the input and output pixel intensities which range from 0 to
255 are scaled to cover the range 0 to 1 .
• Binary - When true or false values are to be used as inputs or outputs
the most straightforward approach is to use 0 for false and 1 for true. Often values of 0. 1 and 0.9 are substituted to avoid some of the inherent limitations in standard back propagation when nodes are near saturation in either direction [Rumelhart & McCleland 1 98 6] .
• Logarithmic compression or scaling - lbis form of pre-processing
allows a large dynamic range to be represented. There is considerable evidence for the use of logarithmically scaled sensory signals in biological systems [Cornsweet 1 970] . Logarithmic scaling also affects the way in which inputs interact in the network. For example addition produces, in effect, a multiplication and subtraction provides a ratio of the raw values.
• Binary encoding - lbis can be used for both inputs and outputs and involves replacing a single connection by a vector each component of which represents one binary bit. Each of these bits is then considered as a separate input or output. Again the binary values may be internally represented by 0. 1 and 0.9 rather than 0 and 1. The discrimination and generalisation required should be considered carefully when using this coding technique as a small change in a parameter value may result in a step change in the binary encoded representation. This may aid or hinder learning. It will be helpful when the values of the parameter do not represent some form of continuum. For example consider a parameter which represents the part number of inventory items which are not ordered into groups. In this case the input for two parts which have adjacent part numbers should be made quite different to aid in discrimination.
• Interval coding - Where the effect of a parameter is known to vary in
different regions of its range it may be advantageous to use a separate input or output for each region. A number of inputs (or outputs) is
Chapter 7
•
Training Heuristics and their Implementation 7 . 8
used to represent the entire range and each unit is active for only part of the range.
Fuzzy Coding - Fuzzy inputs and outputs can often be used to advantage to eliminate the artificially sharp steps between essentially similar inputs or outputs. This can help in generalisation.
When used for input variables the membership functions are constructed for each of the fuzzy concepts and then these are used to transform a single input value into a number of derived values. For outputs a single target value is decomposed in a similar way to provide targets for each of the fuzzy outputs. When running the network the fuzzy output may be used directly or a maximum chosen or some more complex de-fuzzyfication technique used.
7 .2 . 1 .2 network
As mentioned earlier pre-processing can be achieved by using either additional neural networks or algorithmic manipulation of the input variables. Algorithmic pre-processing can be considered as j ust another form of input coding. Logarithmic coding as described above is an example of this. Other examples are the pre-calculation of powers and cross multiples of input variables. A successful application of this technique is described by Tuck [Tuck 1993] .
Another very useful form of pre-processing is the use of self organising networks to extract significant features. Tills is not considered in detail
here but useful examples are provided in the references given
Chapter 7 Training Heuristics and their Implementation 7.9