The thesis constitutes four main chapters while the remaining chapters present the literature review of the subject area and the conclusion of the thesis.
Chapter 2 reviews the literature on the subject. Apart from presenting some of the basic knowledge of both biological and artificial neural networks, it presents some of the works from neuroscience on neural diversity. In particular, it highlights the works which have associated neural diversity with efficiency and increased representational capacity. It also highlights related works from the field of artificial neural networks, particularly on hybrid neural networks, that were motivated by a variety of reasons for optimizing transfer functions.
Chapter 3 explains the nature of the search space of artificial neural networks. It then highlights the related works in transfer function optimization and the motivation for using neural diversity in artificial neural networks. In particular, it shows how transfer function diversity can improve generalization ability in the context literature that links generalization to bias. Specifically, the bias-variance decomposition and the meta-learning theory on the search space of learning algorithms.
Chapter 4 then shows conclusive results of neural diversity leading to a repertoire of accurate computation strategies for neural networks in an ensemble. Neurally Diverse Ar- tificial Neural Networks (NeuDiME) was used to show that neural diversity is also an effec- tive diversity maintenance scheme for ensembles as it exhibits a broad spectrum of accurate computational strategies, which is an essential part of ensembles. The indication of differ- ences in the use of transfer functions for different problems in the results of this chapter paved the way for the next chapter on computational signatures.
Chapter 5 highlights the problem of increased dimensionality of the search space, which results from transfer function optimization. Computational signatures are presented as a means to both initializing and managing the set of transfer functions during optimization, thus strategically overcoming the complexity of the search space. The chapter presents related works from meta-learning. It then highlights how neuroscience-inspired measures can be used to examine artificial neural networks. Furthermore, it shows how the results of the computational signatures produced from the analysis of neural network models with
the neuroscience-inspired measures can be consistent for a problem and discriminatory between different problems. It also shows results of using a graph-based analysis technique developed to study the expected computational strategy for various problems.
Chapter 6 culminates the observations of chapters 3 and 5 into a holistic approach to optimizing artificial neural networks by co-evolving neurally diverse neural networks with their weights and topologies. It highlights the issues of compounding bias and informa- tion transfer and how they can affect generalization ability. The chapter then highlights the framework design and justifications. It also presents the results of using co-evolving neu- rally diverse artificial neural networks with their weights and topologies on 22 benchmarks and compares them to some of the state of the art learning algorithms in addition to learning algorithms such as the MLP, RBF, and SVM (Support Vector Machines). The chapter also compares the results from NeuDiME to the results obtained using the framework. It also highlights interesting results of hybrids and the variance of these learning algorithms such as the MLP-RBF, which consists of a projection layer and a radial basis layer as used in some related works.
Chapter 7 then concludes with the implications of the findings made in the thesis as well as their contributions.
Chapter 2
Background
“These networks represent functions in much the same way that circuits consisting of simple logic gates represent Boolean functions.” - Russel & Norvig (1995)
In this chapter, a brief background is given for the topics involved in the scope of this thesis. Some basic familiarity with neural networks and evolutionary algorithms is assumed; thus, the background does not highlight all the aspects of each topic.
2.1
The Biological Neural Network
In general, the most fundamental unit in a nervous system is a single cell, called the neuron [68]. We know that each neuron is made up of a cell body, also known as the soma, which like other types of cells contains a nucleus within the walls of the cell (i.e. the membrane). There are strands of fibers that extend from this cell body to the proximity of other neurons, known as dendrites. These are responsible for receiving signals from other neurons. Furthermore, each neuron also has a long branch-like fiber that extends farther than the dendrite to the proximity of the dendrites of other neurons, where it fans out in branches. This branch-like fiber, known as the axon, is specialized at conducting a certain type of electric impulse and is insulated by a membrane, known as the myelin sheath. As illustrated in Fig.2.1, the axon is much thicker than the dendrites. Axons conduct electric impulses from somewhere between the cell body and the axon hillock, where the impulses are generated and transmitted through the length of the axon and then along its branches to its tips.
There are various ways in which neurons communicate; the connection illustrated in Fig.2.1 is the tip of each axon ending (or axon terminal) with one neuron being really close to the cell body of another cell, though not quite being in contact. The connection of an
axon terminito a cell body or dendrite is known as a synapse, and the gap in the synapse where the fibres do not quite make contact is called the synaptic cleft. It is known that neurons are not restricted to making connections with neurons alone; they also make con- nections to various other types of cells within the nervous system. Essentially, neurons like the ones illustrated in Fig.2.1 connected in a neighborhood are defined as neural networks.
Figure 2.1: An axon terminus of a neuron connecting to the body of another neuron. (Courtesy of Wikimedia Commons)
Some of the important agents that aid the transmission of signals between cells in the network are neurotransmitters. Neurotransmitters are chemicals stored in vesicles found at axon terminals that are released by the cell at the transmitting side of the synapse (i.e. the pre-synaptic cell) and diffuse to the cell at the receiving end of the synapse (i.e. the post-synaptic cell). It’s release is caused by a series of actions initiated by an electric im- pulse, known as the action potential. The action potential is a type of electric impulse that is characterized by a series of changes in the electric voltage in the axon body which result in spike-like graphs when measured. It is generated as a result of significant stimu- lation caused by the signals of other cells in the network. This results in an impulse being generated between the cell body and the axon of the stimulated cell, which quickly gets conducted along the length of the axon. It then fans out to all the terminals of the axon where it causes a rise in calcium ions (Ca2+) concentration. This in turn causes the vesi- cles containing the neurotransmitters to fuse with the plasma membrane (i.e. the cell wall) leading to them being opened in the direction of the synaptic cleft. The neurotransmit- ters travel to the post-synaptic neuron’s where it increases the electric potential of the cell
at that moment (the membrane electric potential) by changing the ion permeability of the membrane. Given that the cell at the end of the of connection is a neuron, and that the stimulation is sufficient, this will result in the neuron generating an action potential. This process is then repeated at the end of the axon termini of the excited neuron. Depending on the properties as well as the presence of hormones, the response of each neuron will be different [101, 18].
The chemical signals at the post-synaptic neuron are eliminated in various ways; one method is through the use of enzymes attached to the network which destroys the neuro- transmitter [68]; others include either leaving chemical signals to fade away by diffusion or to be taken back by the neuron that initiated the signal.
2.1.1
Learning in Biological Neural Networks
There are a variety of things that aid learning in the brain. As highlighted in the earlier section, these include the use of hormones, enzymes, neurons and various other forms of communication that lead to learning and a coordinated response.
Learning in the brain is a highly complex process. Generally, stimuli from experiences form, reinforce or prune connections between neurons, which results in an adaptation of the neural circuitry [104, 24]. The process of refining connections between neurons is lifelong; the most active connections are continually strengthened, while connections that are relatively inactive get weaker. At some point connections that are not active get pruned. This process results in both specialization and efficiency [104, 24]. This finding influenced the concept of the McCulloch Pitts neuron, which eventually gave rise to the Perceptron.
Other mechanisms that are believed to be important for learning include components within the biological network - such as neurotransmitters, and feedback signals - and ele- ments in the form of behavior, such as sleeping. The neurotransmitter is also a complex signal pre-processor and is crucial for learning and adaptation [104]. Another role of neu- rotransmitters is sending feedback signals, after activation of a post-synaptic neuron. These feedback signals - sometimes in the form of soluble gasses - are sent backward to modify the behavior of the pre-synaptic neuron [104, 24]. In the category of mechanisms that aid learning from a behavioral perspective is sleep, where our brain has been found to strengthen memory, by stimulating itself in the form of dreams [23]. In general, there are various mechanisms both from behavior and within the biological neural network that are related to learning, either in part of full.
In the context of the human brain, learning can be characterized by its developmental stages. At infancy the brain has an almost fully connected set of neurons relative to its adulthood stage [104, 24], which enables it to adapt to almost any environment. To be
specific there are an estimated 15 - 32 billion neurons [24], with each having about 10,000 connections. Towards its later stages, the human brain is shaped and refined by experiences. In hindsight, it is interesting to see how all these neuroscience findings have directed research in the direction of Artificial Neural Networks. The findings of neuroscience and their inspired artificial neural network research can be listed as follows: Firstly, the findings of the Hebbian learning theory and the excitation of neurons by McCulloch-Pitts inspired Perceptrons. Secondly, the finding of hormones that affect the behavior of neighboring neurons inspired ART (Adaptive Resonance Theory) by Grossberg (1976). In our current time, deep learning has been inspired by findings relating to the deep nature of the brain, which has also had a lot of success [14]. Also, findings related to sleeping have also inspired training routines for neural networks that stimulate them with random stimuli to consolidate learned patterns in a sleep-wake learning cycle [51].
In summary, it can be said that neuroscience is likely to play an even greater roles in Ar- tificial Neural Network research as more technologies for studying the brain and unraveling its mysteries are developed.
2.1.2
Neural Diversity in Biology
This thesis presents research in the direction inspired by the neural diversity property [18, 101, 99] of biological neural networks towards artificial neural networks that are more efficient and with better generalization ability.
Neural diversity is one of the characteristic properties found in biological systems; for example, there are no less than twenty classes of interneurons in the hippocampus alone. In areas such as the retina, there are more than fifty neuronal types. And it is not just coincidental, as research shows that neuronal diversity in these systems has some intrinsic advantages [101, 18, 99].
Apart from neurons being distinct by structure, it is also proven that neurons of the same group are also diverse in terms of behavior. Initially, slight differences in the responses of neurons to incoming stimuli were believed to be noise. However, isolating the neurons and testing their responses to stimuli showed that the response of each neuron - though anatomically similar - varied. Further research at Carnegie Mellon University has taken the research further to alter the stimuli delivered to the neurons; the results showed that neurons responded based on the characteristics of the stimuli [101]; for example, while stimuli characterized by rhythms triggered responses in some, it did not in others, and while some responded to stimuli characterized by rapid changes, others did not. As this research explains, the variations in neuronal responses in a group of heterogeneous neurons
are many folds more than those found in a group of homogeneous neurons, which suggests that the variations of this diversity are not just coincidental.
The concluding finding of the work [101] was that neural diversity is essential for the efficient and effective functionality of biological systems such as the brain. Additional re- search by Thivierge [99] suggested that diversity plays complex roles in aspects concerning our representation and interpretation of the world. He further proposed that the diversity of neurons in a network enables a wider variety of responses with less effort.
In summary, neural diversity is another prevalent property of biological neural net- works, whose constituent neurons have shown to vary in their responses even when they belong to the same group. The variance allows for a broader range of output responses from the network than if they were made up of a homogeneous group. In addition, it also enhances efficiency in terms of the relatively fewer number of neurons required, and to improve the effectiveness of neural networks in terms of the representational capacity associated with it.
2.1.3
Multifunctional Neurons
Apart from the diversity between neurons even of the same anatomy, which was highlighted in the previous section, neurons also foster more representational capacity by their ability to be polymorphic.
The early belief about the nervous system was that it stimulates a pool of interneurons, known as command neurons, which were responsible for the intended behavior. Eventu- ally, it was found that interneurons - a class of neurons usually linking neurons to motor neurons, glands, or organs - can exhibit a variety of responses depending on many factors; including synaptic input, modulation state, and plasticity. These neurons are classed as multifunctional neurons [18]. Multifunctional neurons can also, therefore, be regarded as one of the facets of neural diversity. This is because they are not restricted in their behavior, which would have made them strictly homogenous; but they exhibit a variety of behaviors depending on the incoming signal and their individual properties.
There have been efforts to understand this property of neurons better; for example, work by Marder et. al. [69] has shown that interneurons that activated for some motor behaviors such as respiratory gill movements in invertebrates are also activated for other actions such as spontaneous gill movements and sensory induced gill movements. The findings showed that these interneurons are being shared to exhibit various behaviors. Further study by Briggman [18] which focused on the multifunctional nature of neural circuits summarized and inferred from earlier research, the factors that cause these several functional switches.
These factors were reported to include: inputs from sensory or projection neurons, the in- fluence of neuromodulatory substances, and movement constraints on the organism’s body. Two distinct projection neurons responsible for distinct behaviors (i.e. mill rhythms) found in lobsters that were activated in parallel resulted in a switch in the configuration of the network, enabling two distinct rhythms. Similarly sensory neurons, specifically mechanosensory neurons, when activated simultaneously can cause a reconfiguration that can result in different rhythm behaviors [18].
In addition to synaptic inputs, another influencing factor that can cause a neuron to switch between behaviors is neuromodulator type. Neuromodulators are chemical sub- stances that are released either to alter the properties of neurons, thus changes the way they react to stimuli, or to alter the synaptic strength of a population of neurons. The release of the neuromodulator could be targeted at specific cells, thus being released locally by projection neurons or released in a more hormonal pattern to affect the whole population of neurons. When released, these substances can alter the movement in organisms, switch between movements or even create new movements [18].
In summary, multifunctional neurons could be considered a neural feature that enables a smaller population of neurons to carry out tasks that otherwise would have required a lot more neurons. In other words, it can be regarded as an efficiency-enhancing property.