Estado de M´exico Campus
School of Engineering and Sciences
Hardware Implementation of Spiking Neural Networks for Embedded Systems Applications
A thesis presented by
Diego Concepci´on Villatoro Ger´onimo
Submitted to the
School of Engineering and Sciences
in partial fulfillment of the requirements for the degree of
Master of Science
in
Computer Science
Monterrey, Nuevo Le´on, June, 2021
Acknowledgements
I would like to express my deepest gratitude to all those who have been side by side with me, along the long, also to the Tecnol´ogico de Monterrey which support on tuition and CONACyT with the support for living.
v
Embedded Systems Applications Diego Concepci´on Villatoro Ger´onimo by
Abstract
Currently, conventional computer architectures have reached physical limitations that compro- mise the exponential increase of processing speed, as stated in Moore’s Law. Consequently, other areas have been explored to handle massive information and high computational de- mands in a reasonable time. One of these areas is devoted to understanding how the brain works to get similar results in intelligent information processing with low energy consump- tion and high processing speed. Under this context, in this thesis, a configurable multilayer spiking neural network was implemented using different simplification techniques to reduce hardware resource utilization. The proposed implementation is based on the Izhikevich neu- ron model that can exhibit multiple dynamics found in biological neurons. Also, in the interest of getting a hands-on network, we presented a process to create a spiking network on a digital hardware system, improving some existing archetypes. Although there are many areas where time, optimal use of resources, and energy consumption are especially needed due to practi- cality and physical conditions, we focus on embedded applications with certain limitations.
Although the task surpasses us because spiking networks look to break cognitive barriers that limit the current artificial intelligence algorithms, we are confident in contribute to advance technology in this sense compared to the current state of the art, indicating some research directions that could be the best due to they are biologically inspired.
vii
List of Figures
1.1 A brief snapshot of the major advantage, main application area and main prob-
lems to solve from each approach. . . 2
2.1 Nervous system architecture adapted from [67] [54]. . . 8
2.2 Parts of a biological neuron [20]. . . 9
2.3 Neurons classification according to the number of processes originated from the cell body [67]. . . 10
2.4 Arranging of the neurons on the neocortex by distinctive layers [67]. . . 11
2.5 Neuro-computational properties of biological spiking neurons [38]. . . 17
2.6 The six most fundamental classes of firing patterns of neocortical neurons in response to pulses of depolarizing DC. [39]. . . 19
2.7 Basic Architecture of a Feed Forward Architecture [1]. . . 21
2.8 LeNet-5: One of the earliest convolutional neural networks [1]. . . 21
2.9 A simple CapsNet with 3 layers [70]. . . 21
2.10 Representation of a connection for neuron i with 3 connected neurons in a typical feedforward network. . . 22
3.1 The DPI neuron circuit [34]. . . 28
3.2 Address-event representation. Sender events are encoded into an address, sent over the bus, and decoded. Note that the time axis goes from right to left [25]. 31 3.3 STDP characteristic learning curve [80]. . . 32
4.1 Neuron models to approximate 0.04v2+ 5v + 140reduced by dt = 1/4. . . . 48
4.2 Techniques to approximate original model reduced by dt = 1/4. . . 48
4.3 Arithmetic pipeline for u equation using dt = 1/4, a = 5/256 and b = 0.2. . 49
4.4 V equation arithmetic pipeline to original model using dt = 1/4. . . 50
4.5 V equation arithmetic pipeline to approximate original model using dt = 1/4. 50 4.6 V equation arithmetic pipeline for PWL2 using dt = 1/4. . . 51
4.7 V equation arithmetic pipeline for PWL3 using dt = 1/4. . . 52
4.8 V equation arithmetic pipeline for PWL4 using dt = 1/4. . . 52
4.9 V equation arithmetic pipeline for PLE using dt = 1/4. . . 53
4.10 V equation arithmetic pipeline for CORDIC Multiplication using dt = 1/4. . 54
4.11 V equation arithmetic pipeline for CORDIC Hyperbolic using dt = 1/4. . . . 55
4.12 18-bit LFSR. . . 56
4.13 V equation arithmetic pipeline for Stochastic using dt = 1/4. . . 57
4.14 V equation arithmetic pipeline for Stochastic WBG using dt = 1/4. . . 58 ix
techniques from Izhikevich neuron. . . 59
4.18 Neuron models dynamics after optimizing parameters for 5 typical patterns. . 63
4.19 Representation of a connection for spiking neuron i with 3 connected neurons in a feedforward SNN. . . 64
4.20 Neuron types according layer. . . 65
4.21 Input encoding module. . . 66
4.22 AER interconnection module. . . 66
4.23 Weight bank module. . . 67
4.24 Neuron module diagram. . . 67
4.25 VHDL top SNN module. . . 68
4.26 VHDL layer module. . . 69
4.27 AER module. . . 69
4.28 Neuron module. . . 70
4.29 RAM module. . . 70
4.30 Reduced VHDL interconnection block diagram. . . 71
4.31 VHDL simulation for RAM module with preloaded weights. . . 72
4.32 VHDL simulation for AER module available to represent 5 neuron addresses. 73 4.33 VHDL simulation for neuron module for a single times step. . . 74
4.34 VHDL simulation for neuron module for several times steps. . . 74
4.35 VHDL simulation for input encoding module, spiking every 5 neuron times steps. . . 75
4.36 VHDL simulation for layer 0 encoding the input in the multilayer feedforward SNN with random weights. . . 76
4.37 VHDL simulation for layer 1 setup as RS neurons in the multilayer feedfor- ward SNN with random weights. . . 76
4.38 VHDL simulation for layer 2 setup as IB neurons in the multilayer feedfor- ward SNN with random weights. . . 77
4.39 VHDL simulation for layer 3 setup as FS neurons in the multilayer feedfor- ward SNN with random weights. . . 77
4.40 VHDL simulation for layer 4 setup as CH neurons in the multilayer feedfor- ward SNN with random weights. . . 78
4.41 VHDL simulation for layer 5 setup as LTS neurons in the multilayer feedfor- ward SNN with random weights. . . 78
4.42 Implemented STDP learning rule for the parameters A+ = 0.75, A− = 0.75, τ + = 7.0, and τ− = 4.0. . . 79
4.43 Input images 7x5 for the SNN. . . 80
4.44 Network for digits recognition. . . 80
4.45 Simulation for digits pattern recognition using the weights from [53]. . . 81
4.46 Improvement of the simulation on the digit pattern recognition using our train- ing algorithm. . . 81
4.47 Input images 8x8 for the SNN. . . 82
4.48 Network for figures recognition. . . 82 x
4.49 Simulation on the figures recognition case study. . . 83 4.50 Input images 8x8 for the SNN. . . 83 4.51 Network for alternated pixels recognition. . . 84 4.52 Simulation on the alternated pixels recognition case study with multiple dy-
namics within the same network. . . 84 A.1 Signed Fixed Point Representation. . . 103 A.2 Software Analysis to solve equations. . . 105 A.3 Neuro-computational properties adapted to shifting operations and dt = 1/4. 106
xi
2.1 Comparison of the three fundamental neuron models . . . 14
2.2 Comparison between Spiking Neural Networks (SNNs) and traditional Arti- ficial Neural Networks (ANNs) [85] . . . 26
3.1 Comparison of FPGA approaches to SNNs development . . . 36
4.1 The best 10 approaches for the Original Approximation model (Equation 4.6). 45 4.2 The best 10 approaches for the PWL2 model (Equation 4.7). . . 45
4.3 The best 10 approaches for the PWL3 model (Equation 4.8). . . 46
4.4 The best 10 approaches for the PWL4 model (Equation 4.9). . . 46
4.5 The best 10 approaches for the PLE model (Equation 4.12). . . 47
4.6 The best 10 approaches for the CORDIC Hyperbolic model (Equation 4.18). . 47
4.7 Parameters approximation of typical patterns neuron for Izhikevich Model . . 59
4.8 Parameters of typical patterns neuron for the Izhikevich model [37] . . . 60
4.9 Optimization of typical patterns parameters for Exact Equation (Model 1) . . 61
4.10 Optimization of typical patterns parameters for Exact Equation (Model 2) . . 61
4.11 Optimization of typical patterns parameters for PWL2 (Model 3) . . . 61
4.12 Optimization of typical patterns parameters for PWL3 (Model 4) . . . 61
4.13 Optimization of typical patterns parameters for PWL4 (Model 5) . . . 61
4.14 Optimization of typical patterns parameters for PLE (Model 6) . . . 62
4.15 Optimization of typical patterns parameters for CORDIC Mul (Model 7) . . . 62
4.16 Optimization of typical patterns parameters for CORDIC Hyp (Model 8) . . . 62
4.17 Optimization of typical patterns parameters for Stochastic (Model 9) . . . 62
4.18 Optimization of typical patterns parameters for Stochastic WBG (Model 10) . 62 4.19 Configuration of net for exhibit multiple dynamics . . . 75
5.1 Use of resources for the 10 approximations models of Izhikevich Neuron im- plemented in FPGA (Spartan 6 XC6SLX45 CSG324) . . . 86
5.2 Device utilization for the 10 approximations models of Izhikevich Neuron implemented in FPGA (Spartan 6 XC6SLX45 CSG324) . . . 86
5.3 Ranked Models according to MAE, Resources and Speed from best to worse . 87 5.4 State of the Art Comparison for Single Neuron Implementation . . . 88
5.5 VHDL Synthesis on Spartan 6 xc6slx150t-3fgg676 . . . 89
5.6 Comparison of FPGA approaches to SNNs development with our development 90 A.1 Adjusted parameters for dt = 1/4 and t time vector . . . 107
xiii
Contents
Abstract vii
List of Figures xi
List of Tables xiii
1 Introduction 1
1.1 Problem Definition . . . 4
1.2 Hypothesis and Research Questions . . . 5
1.3 Objectives . . . 5
1.4 Thesis Organization . . . 6
2 Theoretical Framework 7 2.1 Biology of the Brain Architecture . . . 7
2.1.1 Biological Neuron . . . 8
2.1.2 Neocortex . . . 10
2.2 Computational Neuron . . . 12
2.2.1 Neuron Math Models . . . 12
2.2.2 Izhikevich Neuron Model . . . 14
2.3 Neural Networks Architectures . . . 19
2.3.1 Interconnection Architectures . . . 20
2.3.2 Further Details for Multilayer Feedforward ANN . . . 22
2.4 Learning for Neural Networks . . . 23
2.5 Beyond Conventional Neural Networks . . . 25
3 State of the Art 27 3.1 Spiking Neuron Implementation . . . 27
3.1.1 Implementations in Software . . . 27
3.1.2 Implementations in Analogue Hardware . . . 28
3.1.3 Implementations in Digital Hardware . . . 28
3.2 Considerations to Develop Feedforward SNN over FPGA . . . 30
3.2.1 Input Encoding to SNN’s . . . 30
3.2.2 Hardware Interconnection for Massively SNN’s . . . 31
3.2.3 Learning Methods in SNNs over FPGA . . . 32
3.2.4 FPGA implementation for SNNs . . . 33
3.3 Neuromorphic Systems Projects . . . 37 xv
4.1.2 Coefficients Tuning Toward Hardware Implementation . . . 44
4.1.3 Discrete Equations and FPGA Implementation . . . 49
4.1.4 Parameters Tuning Toward the Improve of the Dynamics . . . 60
4.1.5 Analysis and Discussion on Approximation Models . . . 63
4.2 Feedforward SNN Design . . . 64
4.2.1 Input, Hidden and Output Neurons . . . 64
4.2.2 Component Schematic for Hardware Design . . . 65
4.3 VHDL Network Model . . . 68
4.3.1 General VHDL Hierarchy . . . 68
4.3.2 Component and Architecture Verification . . . 72
4.4 Case Studies . . . 79
4.4.1 Training Process . . . 79
4.4.2 Digits Pattern Recognition . . . 80
4.4.3 Figures Pattern Recognition . . . 82
4.4.4 Alternated Pixels Pattern Recognition . . . 83
5 Hardware Implementation Results 85 5.1 FPGA Implementation of a Single Neuron . . . 85
5.1.1 Resources Utilization . . . 85
5.1.2 Comparison with the State of the Art . . . 87
5.2 FPGA Implementation of the SNN . . . 89
5.2.1 Resources Utilization . . . 89
5.2.2 Comparison with the State of the Art . . . 89
6 Conclusions and Future Work 91 6.1 Main Contributions . . . 92
6.2 Future Work . . . 92
Bibliography 100 A 103 A.1 Fixed-Point Operations . . . 103
A.2 Software Analysis of Izhikevich Model . . . 104
xvi
Chapter 1 Introduction
Microprocessor-based hardware architectures limit the performance of classical computer al- gorithms of Artificial Intelligence (AI). At some point, inevitably, physical limitations will be reached and cause a significant impact to process information in a reasonable time. More- over, by a constant generation of Big Data, demanding to analyze them has increased, but the technology industry faces theend of Moore’s Law, whose prediction says that the number of transistors on a microprocessor chip will double every two years. That statement is close to ending by physic constraints as the thermodynamical effect, the increasing thermal noise voltage, together with the trouble of using lower supply voltages to keep power dissipation manageable with the increasing clock frequency [43]. Moore’s Law represented the power of technology to increase data process capability without changing the way to design computer architectures. As a result, we need to explore new scopes to discover the next step to ushers in a new computing age. In these fields, we can mention some of the different roads regarding this issue. Also, it is crucial to keep in mind that answer to computational problems could be an integral combination or set of solutions in continuous evolution:
→ 3D Integrated Circuits look to overcome interconnect scaling barriers by leveraging fast, dense interdie vias, thereby providing an opportunity for continued higher perfor- mance [46].
→ GPU Computing has a large community of practitioners where many of them have reported 10 to 100 times their applications’ speedup [57].
→ Quantum Computing has the idea of using quantum mechanics to perform compu- tations instead of classical physics. Thus quantum computers offer an essential speed advantage over classical computers [62].
→ Molecular Computing involves DNA computing; even so, there are other forms, in- cluding membrane calculation, evolutionary calculation, virus calculation, and others.
The core advantage of molecular computing is its potential to pack vastly more circuits on a microchip than silicon will ever be capable of [50].
→ Programmable Microfluidics is a new way in Neuromorphic, designing a neuromimetic architecture of one neuron based on microfluidic techniques. The main advantage of that neurons is the biocompatibility for biological experiments [49].
1
→ Neuromorphic Computing is an area that looks to implement biological solutions many orders of magnitude more effective than those we have been able to implement using traditional methods [56].
Figure 1.1 shows the significant advantage and the main application area of each ap- proach; however, the options are still open to researchers. Regardless of choice, there are different problems to solve to a greater or lesser degree to go further in computer science, especially to overcome the current power to solve a computational task. Literature supports each option and encourages the continuing discovery of new techniques and applications.
Consequently, for this thesis, we choose to focus onNeuromorphic Computing because it offers the potential cognitive process required in neural networks and supports developing new architectures that leverage the native hardware resources. Also, the human brain could be the most complex and exciting organ to emulate. The human brain contains an estimation of 100 billion neurons, making together roughly 100 trillion synapses [84] with an efficient and close to optimal performance to complete any cognition process with an energy consumption of about 20% of the total [75]. The estimated average ingestion is 2200 kcal per day, which means brain energy required represents around only 20 Watts, less than most of the bulbs consume in our houses. In this sense, neuromorphic computing would be the notable bet to understand the brain and apply this knowledge to solve real problems, mainly related to the so-called intelligent information processing.
Figure 1.1: A brief snapshot of the major advantage, main application area and main problems to solve from each approach.
In the beginning, from [58], Neuromorphic Computing is defined as the implementa- tion of biological neural networks as analog or digital copies on electronic circuits. This ap- proach’s goal is twofold: (1) in neuroscience offers a tool to understand the dynamic processes
3
of learning, (2) and in computing science offers the potential of applying brain inspiration to generic cognitive computing. Although both goals are firmly related, we explored the embed- ded applications as an AI system due to improvement in energy efficiency, execution speed, robustness against local failures, and the ability to learn compared to traditional approaches.
As a result, we can see in the current efforts to achieve optimal AI algorithms that each step attempts to get closer to understand how the brain works, but it remains the enigma. In this sense, for AI, there are three differentiated generations to the present:
→ The first generation of AI was rules-based and emulated classical logic to draw rea- soned conclusions within a specific, narrowly defined problem domain. It was well suited to monitoring processes and improving efficiency, for example [35].
→ The second and current generation is largely concerned with sensing and perception, such as using deep-learning networks to analyze the contents of a video frame [35].
→ The next third generation using a biological approach intent to extend AI into areas corresponding to human cognition, being optimist, this is critical to overcoming the so- calledbrittleness of AI solutions based on neural network training and inference, which depend on literal, deterministic views of events that lack context and commonsense understanding [35].
Artificial Neural Networks (ANN) and Machine Learning (ML) algorithms’ limitations implemented on conventional computer architectures are the central motivation of this text in a search to break out restrictions and continuously improve IA algorithms in computer science.
It is precisely in the third generation of AI where neuromorphic applications can be founding.
Below are listed the main points as motivation for this work, and although neuromorphic engineering is a complex path, it has a strong history that supports it.
→ Uncover the brain science mysteries about how it works to solve cognitive activities in deep.
→ Going beyond current physical boundaries for computation to extend it and break paradigms as a programmer, being efficient in resources.
→ Boost the applications to manage large data in real-time, reducing the loss of informa- tion by speed.
Exploring the core of the process information principles will open access to building brain models more efficiently to improve AI algorithms, with several applications in any area that processes data. However, only a reduced target as building a neural processor to embed- ded systems is a complex task. Although it would be one of the best results, the human brain’s emulation involves many resources for which current platforms are not suitable, also several other issues in each step. As a result, with the time and resources available for this thesis work, we designed a spiking neural network to solve a specific problem for embedded applications given a proper configuration. It is our first attempt and the core to go further in a more robust project that allows us to reach a real problem that anyone can solve by apply- ing this technology for a specific area like the internet of things, computer vision, or pattern recognition to solve problems in agriculture, home, medicine, or any other field.
1.1 Problem Definition
We identified some statements to keep working in the present writing about neuromorphic en- gineering; however, there are several issues, some of those mentioned in Figure 1.1. There are substantial technological advances to emulate the brain works, but a complete and accessible solution is not available yet to computation’s actual troubles. Likewise, we focus on three main problems in this first attempt to design neuromorphic hardware.
→ Complex Design. A significant challenge to design neuromorphic is to provide an am- pleconfigurability and scalability easy to use. However, the cost of freely configurable connections growing fast with the number of neurons in the system [64], and as we are interested in embedded systems, that is a complex balance between precision and op- timality. Notwithstanding, we need to search for simplification techniques to improve the use of a selected platform’s resources, reducing the precision loss impact.
→ Massive Interconnection. Even if we have a neuron model design, we need to in- terconnect the neurons for a particular topology, which requires many physical wires.
As we previously mentioned, the human brain comprises approximately 100 billion elementary processing units called neurons, and each neuron connects approximately 1000–10000 other neurons through synaptic connections. Emulate those actual quan- tities results in an almost impossible task at the current days. Consequently, for this reason, one of the main problems is the implementation of optimal parallel hardware for large sets of neural populations. Besides, network emulations are an abstraction of the brain three-dimensionality in a 2D design. In a parallel 2D hardware, the physical wiring only allows implementing connections between neighboring neurons [11]. Un- derstanding the brain with a straightforward model is a significant challenge, but it is necessary to implement an interconnection scheme.
→ Impracticality Applications. Neuromorphic systems involve change code paradigms in computation design to being useful in real problems because, at this moment, there is not a generalized way to solve a vast amount of problems. There are strong efforts as TrueNorth, one of the most popular neuro-processor, which requires a very different programming model than more traditional CPUs or graphics processing units; their architecture does not have an addressable central memory from which to fetch, decode, and execute instructions for its neurons [19]. Even though the progress, the door is still opened to improve a suitable environment to test a neuromorphic platform. Moreover, when we talk about embedded application, there are essential limits inenergy, speed, andoccupied surface that it is crucial to accomplish if we want to solve real problems.
If we continue exploring Neuromorphic Engineering, we will found more enigmas. The above listed the main objective problems in this thesis document to design neuromorphic sys- tems. However, we only covered a small piece of these troubles through simplified models, but we want to investigate an alternative computation on hardware applied to embedded systems.
Furthermore, Neuromorphic opens a palette of tones because there are many fields where dramatic improvements over AI algorithms can be applied, such as autonomous navigation, smartphones, the internet of things, or control systems.
1.2. HYPOTHESIS AND RESEARCH QUESTIONS 5
1.2 Hypothesis and Research Questions
What are the core principles to process information in the way that the brain works? Or how does the brain work to be efficient in a reasonable time in terms of energy and speed?
Those previous questions could be some of the most critical questions and motivations in neuroscience. However, in this work, to contribute to this endeavor, we focus on particular but still interesting questions described below. Plus, we keep in mind that throughout the development of this thesis work, we look for profound knowledge answers, leaving the chance open to emerging more interrogatives but trying to contribute with the discussion of results and conclusions:
→ What strategy would be better to implement Spiking Networks to being flexible in hard- ware to embedded applications?
→ How important would be the platform choice to emulate Spiking Networks?
→ How scalable and flexible would be the elaboration of the massively interconnected neurons to embedded applications?
→ What are the main constraints to apply Spiking Networks in embedded applications?
In this thesis work, we are interested in implement an efficient spiking neural network, so we define the following hypothesis to demonstrate through the elaboration of this document:
Using the Izhikevich neuron model and a parallel model oriented to FPGA is a viable and feasible option to design a spiking neural network architecture adhered to biology being scalable and practical to embedded systems comparable to brain works in terms of speed and resources.
1.3 Objectives
Looking to prove the hypothesis, this work’s general objective is:
• To develop a spiking neural network architecture in a range of thousands of neurons, running in megahertz based on Izhikevich neuron model to be scalable and applicable to Embedded Systems.
The network needs to receive external inputs and support a flexible and practical environment over an FPGA chip capable of massively processing parallel. In this sense, to reach the main target of research work, the particular practical goals are:
→ To design a simplified Izhikevich neuron model biological plausible on hardware using VHDL model and FPGA technology.
→ To design architecture to emulate massively interconnected neural networks in the range of thousands to be configurable, with high process speed.
→ To design a software-hardware environment that allows us to train the net in software and synthesize hardware over an FPGA platform to embedded applications.
Additionally, it will be necessary to test neural network designing in case of studies, because further of accomplishing previous goals, we look for leverage the biological approach given by the spiking networks, in this sense, goals are:
→ To open up the possibility to leverage biological approach in cognitive activities to process data.
→ To provide foundations to build technology-based neural systems that can be applied to solve problems as a feasible and viable option.
→ To test FPGA technology to develop spiking neural networks.
There is a wide range of applications to improving current solutions given by AI algo- rithms that they can not solve efficiently. Spiking neurons present a powerful unexplored tool with unknown limits as the brain. We are confident in contributing to this research area, trying to solve the previous goals, and opening new possibilities to other researchers to look for new targets and opportunity areas.
1.4 Thesis Organization
This thesis is structured in 6 chapters starting with the present introductory Chapter 1, then:
• Chapter 2 describes the whole Theoretical Framework from biology support to the primary considerations and concepts to build a spiking neural network knowing the maths behind biology. The biology part is a general description of the brain structure to understand the bases of emerging spiking networks; then, the dynamics and features exhibited by a computational neuron; and finally, a review of artificial networks.
• Chapter 3 reviews the State of the Art in this topic to show the efforts of researchers to develop multiples techniques in spiking nets, from the neuron to network, including the results of researchers in FPGA until some examples of giant companies whose specific implementation represents a solid argument to continues working in this area.
• Chapter 4 reviews the Proposed Hardware Implementation explaining the experi- mental process to achieve the previously defined goals and highlighting the most impor- tant problems to focus the subsequent efforts in this area, from the neuron to network, including the experimental process of applying in the case studies.
• Chapter 5 reviews the Hardware Implementation Results over FPGA synthesis com- pared with the current State of the Art. We choose specific reported results from the literature, comparing multiple works for making a better analysis.
• Finally, in Chapter 6 of Conclusions, we analyze the global results, describing this thesis’s main contributions and the expected future work.
Chapter 2
Theoretical Framework
The performance of traditional computer architectures is facing the end of Moore’s Law;
wherefore, it is valid to look at other horizons. In response, for years, computer science has tried to emulate the brain’s works to code and process massively information; nevertheless, today is a great mystery box that is hard to duplicate. Moreover, there are many cognitive activities only a human could do in current days, making AI algorithms seem like a bit of a child. Looking to understand brain works, we start by reviewing the biological inspiration attempts to see a snapshot of the complexity inside the brain and its main structure, also get the idea about the motivations behind how the researchers look to emulate the brain or part of it getting clues of the efficiency which make at this organ the more powerful computational unit. Then a single neuron analysis to emulate it computationally from a biology approach, followed by neural network structures and considerations to design them to solve real prob- lems. Finally, a recap of the potential of bio-inspired systems in any area beyond simple math operations.
2.1 Biology of the Brain Architecture
The human brain is a network of more than 100 billion individual nerve cells interconnected in systems that construct our perceptions of the external world, fix our attention, and con- trol the machinery of our actions [67], reason because it is the more great organ conceived in our body. After a literature review, we use Figure 2.1 to show a simplified schema of the nervous system, focusing on the brain. However, it is essential to consider that it is not the only anatomical classification, and it could be arranged differently depending on the refer- ence. Moreover, in the brain, we focus on the cerebral cortex, which appears to be the most promising candidate for determining the direct neuroanatomical correlations of intelligence [48]. Nevertheless, analyzing this tiny part of the brain continues to be an extraordinary task.
We do not expect to deepen in each part of the nervous system; only we want to exemplify the complexity behind taking a small brain piece of this arrangement, as the neocortex to analyze it, and although we only expect to create an extraordinarily simplified model of that piece, it could arrive at critical applications and improvements over resource management. Besides, we could improve multiple computational tasks, considering the same physical constraints in the conventional architectures.
7
Figure 2.1: Nervous system architecture adapted from [67] [54].
2.1.1 Biological Neuron
The brain units,neurons, are fundamental pillars in charge of transmitting information to the body; they have a complex way to codify information such as the firing spike rate, the neuron as population, and the way of the spikes. This neuronal code makes it possible to move, keep reminders, execute actions, reason, or think. Despite all information, the way to make it possible is a mystery yet, maybe as a single cell look simple but the join of millions creates ourmind.
A single neuron is formed bydendrites, axons, and cellular body, as we see in Figure 2.2. The cellular body and dendrites receive the input from other neurons; meanwhile, the axon transmits the output to other neurons taking advantage of his grease insulating layer covered by myelin to improve it. Besides, a typical neuron has thousands of dendrites and axon branches in its end so that it may affect thousands of others through small structures named terminal buttons. Moreover, the connection between neurons is called a synapse, and the space existing in the synapse is designated synaptic cleft [20]. Although Figure 2.2 shows a traditional image of the neuron, there are several ways to classify them, for instance, according to their function, morphology, or response.
By functions, we found: sensory neurons, motor neurons, and interneurons [67].
→ Sensory neurons carry information from the body’s peripheral sensors into the nervous system. Sensory neurons are also called afferent neurons.
2.1. BIOLOGY OF THE BRAIN ARCHITECTURE 9
→ Motor neurons carry commands from the brain or spinal cord to execute actions in muscles and glands. Motor neurons are also known as efferent neurons.
→ Interneurons are the most numerous and are subdivided into two classes: relay and local. Relay or projection interneurons have long axons and convey signals over con- siderable distances, from one brain region to another. Local interneurons have short axons because they form connections with nearby neurons in local circuits.
Figure 2.2: Parts of a biological neuron [20].
Another way to classified neurons is by their morphology into three large groups [67]:
→ Unipolar neurons are the simplest because they have a single primary process, which usually gives rise to many branches. One branch serves as the axon; other branches function as receiving structures (Figure 2.3 A). These cells predominate in inverte- brates’ nervous systems; in the vertebrates are present in the autonomic nervous system.
→ Bipolar neurons have an oval soma that gives rise to two distinct processes: a dendritic structure that receives signals and an axon that carries information toward the central nervous system (Figure 2.3 B). Many sensory cells are bipolar, including those in the retina and the olfactory epithelium of the nose. The receptor neurons that convey touch, pressure, and pain signals to the spinal cord, are variants of bipolar cells called pseudo- unipolar cells. They start as bipolar cells, but the two cell processes fuse into a single continuous structure (Figure 2.3 C).
→ Multipolar neurons predominate in the nervous system of vertebrates. They typically have a single axon and many dendritic structures emerging from various cell body points (Figure 2.3 D). Multipolar cells vary greatly in shape, especially in the length of their axons and in the extent, dimensions, and intricacy of their dendritic branching. Spinal motor neurons can receive around 10,000 contacts. The dendritic tree of a Purkinje cell in the cerebellum is much larger, receiving up to million contacts. Pyramidal cells have a roughly triangular cell body; dendrites emerge from both the apex (the apical dendrite) and the base (the basal dendrites). These neurons are found in the hippocampus and throughout the cerebral cortex.
Figure 2.3: Neurons classification according to the number of processes originated from the cell body [67].
There are millions of neurons; even inside each classification described, there are several variations. A complete classification in current days continuous being one challenging neuro- science target. Looking for simplifying the classifications, we only need to keep in mind that the bipolar neurons usually are sensory neurons, and interneurons and motor neurons usually are multipolar, although the reality is far more complex. It is similar to artificial networks, which typically are organized in input, hidden, and output neurons.
2.1.2 Neocortex
After a brief introduction to neurons, we continue with the brain’s primary interest area to analyze. The neocortex receives inputs from the thalamus, other cortical regions, and other brain structures, and its output connects to other regions of the neocortex, basal ganglia, thala- mus, pontine nuclei, and the spinal cord. These complex input-output relationships allow for an orderly and efficiently organized layering of cortical neurons [67]. Most of the neocortex contains six layers, numbered from the outer surface (pia mater) of the cortex to the white matter as Figure 2.4 shows containing up to 28x109 neurons with around 1012synapses [79].
→ Layer I is occupied by the dendrites of cells located in deeper layers and axons that travel through this layer to make connections in other areas of the cortex [67].
→ Layers II and III contain mainly small pyramidal-shaped cells. Layer II is one of two layers that contain small spherical neurons. In Layer III the neurons located deeper are typically larger than those located more superficially. The axons of pyramidal neurons
2.1. BIOLOGY OF THE BRAIN ARCHITECTURE 11
in layers II and III projects locally to other neurons within the same cortical area as well as to other cortical areas, thereby mediating intracortical communication [67].
→ Layer IV contains a large number of small spherical neurons. It is the main recipient for sensory input from the thalamus and is most prominent in primary sensory areas.
For example, the occipital cortex region that functions as the primary visual cortex has an extremely prominent layer IV. Layer IV in this region is so heavily populated by neurons and so complex that it is typically divided into three sublayers [67].
→ Layer V contains mainly pyramidally shaped cells that are typically larger than those in layer III. Pyramidal neurons in this layer give rise to the cortex’s major output pathways, projecting to other cortical areas and subcortical structures [67].
→ The neurons in layer VI are fairly heterogeneous. It blends into the white matter that forms the cortex’s deep limit and carries axons to and from areas of the cortex [67].
Figure 2.4: Arranging of the neurons on the neocortex by distinctive layers [67].
The neocortex results in one of the brain’s most exciting areas and a challenge to under- stand, but the award is a fantastic science achieve- ment. It is fascinating to observe the neocor- tex arrangement because we do not have a com- plete idea about why this layered scheme and why these layers amount, but it works. Again, the re- ality is far more complex because the neocortex is not a homogeneous structure; it is just a conven- tion to distinguish six layers. Furthermore, the actual function of cortical lamination is unknown;
even the evidence presented in [29] suggests that the function does not reside in the layer but the circuit, irrespective of its specific spatial layout.
Thus, there are several factors to consider if we desire to emulate the neocortex’s biology. Such is the case that we only limit to keep in mind that the idea of six layers appropriately interconnected is at least enough to solve complex tasks. Con- cluding, it was only a resume of some key con- cepts from biology, and it is essential to be care- ful with the information due to changes through neuroscience evolution.
2.2 Computational Neuron
After looking at a biological summary, we start with the first challenge of creating a single unit to be practical in computer science; in this case, we refer to neurons. To understand neurons working in the brain articulated as a whole, multiple math models to emulate them were developed through the years, improving biological response, computational efficiency, or freely parameters adjustment according to the work requirements. In this case, we look for hardware development, so we need to consider certain compromises concerning biological plausibility, computational complexity, and flexibility, but it is not a thorough revision, so we take previous works analysis.
2.2.1 Neuron Math Models
We present three of the most fundamental computational neuron models, although existing a wide variety that could be analyzed or studied depending on the case and specifications:
→ Integrate & Fire
One of the most widely used models in computational neuroscience is the leaky integrate- and-fire (I & F) neuron, described below:
v� = I + a− bv (2.1)
If v ≥ vthresh, then v ← c, being v the membrane potential, I is the input current, and a, b, and c the parameters. When the membrane potential v reaches the threshold value vthresh, the neuron is said to fire a spike and is reset to c. In summary, despite its sim- plicity, Izhikevich demonstrates in [38] its simplicity limits their biological response, making at I&F one of the worst models to use if we want biological realism.
→ Spiking Model by Izhikevich
Izhikevich proposes his neuron model more complex than I&F, using two differential equations instead of one but with more biological plausible responses:
v� = 0.04v2+ 5v + 140− u + I (2.2)
u� = a(bv− u) (2.3)
with the auxiliary after-spike resetting if v ≥ +30mV , then v ← c and u ← u + d.
Here variable v represents the membrane potential of the neuron and u represents a membrane recovery variable, which accounts for the activation of K+ ionic currents and inactivation of Na+ ionic currents, and it provides negative feedback to v. After the spike reaches its apex (+30mV ), the membrane voltage and the recovery variable are reset according to the third equation. If v skips over 30, then it first is reset to 30, and then to so that all spikes have equal magnitudes. The threshold value of the model neuron is between –70 and -50, and it is dynamic, as in biological neurons [38].
2.2. COMPUTATIONAL NEURON 13
→ Hodgkin–Huxley
The Hodgkin–Huxley model is one of the indispensable models in computational neu- roscience. It consists of four equations and tens of parameters describing membrane potential. Also, the model has quite limited behavior for the original values of parame- ters, and we need to tune them to achieve an optimal result. In this sense, the primary issue is that the model is computational costly to implement due to the equations and parameters; some authors have also described it as analytically impossible, inherently complex, and only viable for small neural networks [76]. On the other hand, their pa- rameters are biophysically meaningful and measurable; consequently, they allow us to investigate questions related to synaptic integration, dendritic cable filtering, effects of dendritic morphology, the interplay between ionic currents, and other related issues to single-cell dynamics [38]. Unless the time and resources do not be an issue or some simplification technique has been applied, this model is not the best option for practical uses, and for that reason, we omit a deep description, but it remains an essential model.
Some other models described in [38] are listed below, but we need to be careful accord- ing to our requirements in response to the particular task that we want to develop because they could be counterproductive for some purposes; however, it is essential to be knowledgeable about their existence.
→ I & F with Adaptation. It is an improvement of the I&F model, which cannot burst or have other properties of cortical neurons, however, the model still lacks many important properties.
→ Integrate-and-Fire-or-Burst. It is an improvement model (I&FB) with interesting features but it has a deficit of speed compared with predecessors.
→ Resonate-and-Fire. The resonate-and-fire neuron is a two-dimensional (2-D) analog of the I&F neuron, simple and efficient for specific tasks.
→ Quadratic I&F. An alternative to the leaky I&F neuron is the quadratic I&F neuron because has spike latencies, activity-dependent threshold, and bistability of resting and tonic spiking modes.
→ FitzHugh–Nagumo. Model is a 2-D system of Ordinary Differential Equations, but, it cannot exhibit autonomous chaotic dynamics or burst.
→ Hindmarsh–Rose. Depending on their parameter choice, the model can, in principle, exhibit optimal response. The problem is, how to find the functions of the model.
→ Morris–Lecar. It is a simple 2-D model to describe oscillations in barnacle giant mus- cle fiber. Because it has biophysically meaningful and measurable parameters. In a certain setting, the model becomes equivalent to the Hodgkin–Huxley model.
→ Wilson Polynomial Neurons. The model uses polynomial equations and it can exhibit all neurocomputational properties, by the proper choice of the parameters, which is not an easy task.
Taken into account the main results from Izhikevich [38], his model is pretty close to optimal, comparing their possible responses and execution time with other models, making a good trade-off between biology, complexity, and flexibility: (1) from biology exhibit 20 computational properties (described later), meanwhile I&F only three features, (2) the number of floating points operations to simulate 1ms that need Izhikevich are 13; meanwhile, I&F needs five, and Hogking-Huxley needs 1200, (3) Hodgkin-Huxley can exhibit a vast set of properties; even more than features described by the author and it is biophysically meaningful, but tune all its parameters involve another complexity degree that limits the flexibility of the model. Table 2.1 resumes the previous points.
Table 2.1: Comparison of the three fundamental neuron models Model Biophysically
Meaningful Biological
Features # of Differential
Equations Parameters # of FLOPS (1ms)
I&F - 3 1 3 5
Izhikevich - 20 2 4 13
Hogking-Huxley + 20 4 tens 1200
Finally, we must choose the best according to the specific problem that we want to solve, e. g., in neuroscience, if we want to understand biological reactions, Hodgkin–Huxley is the best, but the time expensive makes it obsolete to embedded applications. On the other hand, I&F is the simplest, but it lacks a range of biological features. At this point, although the Izhikevich model is not optimal, achieving a solid performance according to response in different properties attached to biology in reasonable execution time [38]. In the next part, we detail this selection explained the properties and dynamics exhibited by the Izhikevich model.
2.2.2 Izhikevich Neuron Model
There are several neuron models, and the literature only has explored a small quantity in deeper experimentation; in this sense, we choose the Izhikevich model because the literature sustains the selection as we described above. To get a more profound knowledge of this model, we review the model’s biological features and dynamics reported by the author, Izhikevich.
Neuro-Computational Features exhibited by Izhikevich Model
Figure 2.5 show 20 of the most prominent features of biological spiking neurons described by Izhikevich that also his model can replicate [38], and they are part of the entire dynamic of the neuron. Below we describe briefly each one:
→ Tonic Spiking. Most neurons are excitable; that is, they are quiescent. This feature consists of the neuron can fire spikes when stimulated. While the input is on, the neuron continues to fire a train of spikes (Fig. 2.5(A)).
→ Phasic Spiking. In this feature, the neuron may fire a single spike at the onset of the input and remain quiescent afterward (Fig. 2.5(B)). Phasic spiking is helpful for the detection of the beginning of stimulation.
2.2. COMPUTATIONAL NEURON 15
→ Tonic Bursting. This feature consists of fire periodic bursts of spikes when stimulated (Fig. 2.5(C)). The interburst (i.e., between bursts) frequency may be as high as 50 Hz, and it is believed that such neurons contribute to the gamma-frequency oscillations in the brain.
→ Phasic Bursting. Similar to the phasic spikers, some neurons are phasic bursters (Fig.
2.5(D)). Such neurons report the beginning of the stimulation by transmitting a burst.
There are three extended hypotheses on the importance of bursts in the brain which are:
1) bursts are needed to overcome the synaptic transmission failure and reduce neuronal noise; 2) Bursts can transmit saliency of the input because the effect of a burst on the postsynaptic neuron is stronger than the effect of a single spike; 3) bursts can be used for selective communication between neurons.
→ Mixed Model (Bursting Then Spiking). It is a mixed type of spiking activity (Fig.
2.5(E)). They fire a phasic burst at the onset of stimulation and then switch to the tonic spiking mode.
→ Spike Frequency Adaptation. This feature consists of fire tonic spikes with decreasing frequency (Fig. 2.5(F)). That is, the frequency is relatively high at the onset of stimu- lation, and then it adapts. The interspike frequency of such cells may encode the time elapsed since the onset of the input.
→ Class 1 Excitability. The frequency of tonic spiking neurons depends on the input’s strength, and it may span the range from 2 Hz to 200 Hz, or even greater (Fig. 2.5(G)).
This feature can fire low-frequency spikes when the input is weak. Class 1 excitable neurons can encode the strength of the input into their firing rate.
→ Class 2 Excitability. This feature consists of neurons that cannot fire low-frequency spike trains. They are either quiescent or fire a train of spikes with a specific relatively large frequency, say 40 Hz 2.5(H)). Their firing rate is a poor predictor of the strength of stimulation.
→ Spike Latency. Most cortical neurons fire spikes with a delay that depends on the strength of the input signal. For a relatively weak but superthreshold input, the delay, also called spike latency, can be pretty large (Fig. 2.5(I)). Some neurons in the mam- malian cortex can have latencies of tens of ms. Such latencies provide a spike-timing mechanism to encode the strength of the input.
→ Subthreshold Oscillations. Practically every brain structure has neurons capable of exhibiting oscillatory potentials (Fig. 2.5(J)). The frequency of such oscillations plays an important role, and such neurons act as bandpass filters.
→ Frequency Preference and Resonance. Due to the resonance phenomenon, neurons having oscillatory potentials can respond selectively to the inputs having frequency con- tent similar to the frequency of subthreshold oscillations. Such neurons can implement frequency-modulated (FM) interactions and multiplexing of signals. In Fig. 2.5(K), the neuron responds only to the doublet whose frequency resonates with the frequency of subthreshold oscillations.
→ Integration and Coincidence Detection. Neurons without oscillatory potentials act as integrators. They prefer high-frequency input; the higher the frequency more likely they fire (Fig. 2.5(L). It can be useful for detect coincident or nearly coincident peaks.
→ Rebound Spike. When a neuron receives and then is released from an inhibitory input, it may fire a post-inhibitory (rebound) spike (Fig. 2.5(M)). This phenomenon is related to the anodal break excitation in excitable membranes. Many spiking neurons can fire in response to brief inhibitory inputs, thereby blurring the difference between excitation and inhibition.
→ Rebound Burst. In this feature, some neurons, including the thalamocortical cells, may fire post-inhibitory bursts (Fig. 2.5(N)). It is believed that such bursts contribute to the sleep oscillations in the thalamocortical system.
→ Threshold Variability. A common misconception in the artificial neural network com- munity is that spiking neurons have a fixed voltage threshold. It is well-known that biological neurons have a variable threshold that depends on the prior activity of the neurons. In Fig. 2.5(O), first, stimulated a neuron with a brief excitatory pulse of cur- rent that produces 10 mV depolarization. The neuron did not fire; hence, the input is subthreshold. Then, it applied a brief inhibitory input and then the same ”subthreshold”
pulse of current. The neuron fired the second time because the preceding inhibitory input lowered its ”threshold”.
→ Bistability of Resting and Spiking States. Some neurons can exhibit two stable modes of operation: resting and tonic spiking (even bursting). An excitatory or inhibitory pulse can switch between modes (Fig. 2.5(P)). Notice that switching from the tonic spiking to resting mode emphasizes the importance of spike-timing in information processing.
→ Depolarizing After-Potentials After firing a spike, the membrane potential of a neuron may exhibit a prolonged after-hyperpolarization (AHP) as, e.g., in Fig. 2.5(B), (I), or (M), or a prolonged depolarized after-potential (DAP), as in Fig. 2.5(Q). Those neurons have shortened refractory period, and it becomes super excitable.
→ Accommodation Those Neurons are susceptible to brief coincident inputs but may not fire in response to a substantial but slowly increasing input (Fig. 2.5(R)). The slowly ramped current in the figure does not elicit a spike, while a smaller but sharply ramped current elicits a spike.
→ Inhibition-Induced Spiking A bizarre feature of many thalamocortical neurons is that they are quiescent when there is no input but fire when hyperpolarized by an inhibitory input or an injected current (Fig. 2.5(S)).
→ Inhibition-Induced Bursting Instead of spiking, a thalamocortical neuron can fire tonic bursts of spikes in response to a prolonged hyperpolarization (Fig. 2.5(T)). It is believed that such bursting takes place during spindle wave oscillations in the thala- mocortical system, and it plays an essential role in sleep rhythms.
2.2. COMPUTATIONAL NEURON 17
Figure 2.5: Neuro-computational properties of biological spiking neurons [38].
Dynamic Types: Typical Patterns in Neurons
From the set of features, although the model should not exhibit all properties simultaneously, when are combined two or more, we can obtain some neuron dynamics; in a certain sense, dynamics are more complex than features. Notwithstanding, it is a challenge to integrate a general classification for these patterns neurons across different laboratories and studies of the world. The more generally accepted classification taking the most significant firing patterns for neocortical neurons in the mammalian brain divide neurons into excitatory and inhibitory. Excitatory neurons incite to spike the connected neurons; meanwhile, inhibitory neurons prevent subsequent spikes in the net. As well as Izhikevich Model can exhibit a specific feature, the model can emulate a firing pattern or dynamic as we describe below.
Excitatory cortical cells can be divided into the following three main classes:
→ Regular Spiking (RS) neurons (Figure 2.6) are the most typical neurons in the neocor- tex. This neuron can exhibit tonic spiking, spike frequency adaptation, spike latency, and class 1 excitability features. In the Izhikevich model, this corresponds to c = 65mV (deep voltage reset) and d = 8 (large after-spike jump of u [37]). The spike frequency of RS cells is fairly linear as a function of injected current (f–I curve) with frequencies of up to 200–300 Hz [16].
→ Intrinsically Bursting (IB) neurons (Figure 2.6) fire a stereotypical burst of spikes followed by repetitive single spikes equal to the mixed model feature. In the model corresponding to c = −55mV (high voltage reset) and d = 4 (large after-spike jump of u) [37]. The burst of spikes at the beginning is around 3-5 action potentials at 200 Hz, followed by tonic spiking, usually at frequencies around 5–15 Hz [16].
→ Chattering (CH), also known as Fast Rhythmic Bursting) neurons (Figure 2.6) can fire stereotypical bursts of closely spaced spikes equal to the tonic bursting feature. In the model, this corresponds to c = 50mV (very high voltage reset) and d = 2 (moderate after-spike jump of u) [37]. CH bursts contain 2–5 spikes at high frequencies (usually from 200 up to 600 Hz), and during sustained depolarization, bursts repeat regularly at frequencies of 20–80 Hz [16].
On the other hand,inhibitory cortical cells have wildly diverse classes; even some of the most typical classes described below have subclasses making it harder to take a representative group of inhibitory neurons that can exhibit enough features from biology:
→ Fast Spiking (FS) neurons (Figure 2.6) can fire regular trains of action potentials with extremely high frequency practically without any adaptation (slowing down). This neu- ron can exhibit tonic spiking and class 2 excitability features. In the model, this corre- sponds to a = 0.1 (fast recovery) [37]. It can generate regular trains of action potentials at very high frequencies (up to 800 Hz) with virtually no accommodation [16].
→ Low-Threshold Spiking (LTS) neurons (Figure 2.6) can fire high-frequency trains of action potentials, but with a noticeable spike frequency adaptation, also they exhibit the tonic spiking feature. These neurons have low firing thresholds, which is accounted for by b = 0 : 25 in the model [37]. The small burst of 2 or 3 spikes riding on a small depolarizing hump allows recognizing these neurons [16].
→ Late Spiking (LS) neurons (Figure 2.6) can show sustained firing during the current step, often exhibit anti-accommodation early in the spike train [5]. LS neurons exhibit a voltage ramp in response to injected Direct Current (DC), resulting in delayed spiking with latencies as long as 1 sec. There is a pronounced subthreshold oscillation during the ramp, but discharge frequency is less than FS neurons [39].
The author did not give the last parameters for the LS neuron dynamic, but it remains an essential pattern for this classification. What is known is that the Izhikevich model also can reproduce behaviors of ThalamoCortical (TC) neurons, which provide the principal input to the cortex; his neurons have two firing regimes: When at rest (v is around −60mV ) and then depolarized, they exhibit tonic firing. However, if a negative current step is delivered, the membrane potential is hyperpolarized (v is around −90mV ), causing neurons to fire a rebound burst [37]. To conclude, Izhikevich Model reproduces a rich biological neurons’ behavior, such as bursting, mixed-mode firing patterns, post-inhibitory (rebound) spikes, and bursts, continuous spiking with frequency adaptation, spike threshold variability, bistability of resting and spiking states, among others [37]. The set of features and dynamics exhibited by the Izhikevich model gives us enough reasons to discuss and analyze this model’s implementation in-depth for embedded applications.
2.3. NEURAL NETWORKS ARCHITECTURES 19
Figure 2.6: The six most fundamental classes of firing patterns of neocortical neurons in response to pulses of depolarizing DC. [39].
As we are looking to create a network, we need to go beyond the specific features de- scribed formerly. We need to replicate the dynamics, which are the natural patterns in the brain. In this sense, the valuable information provided by Izhikevich author allows us to sim- ulate at least five neocortical dynamics: three excitatory patterns (RS, IB, and CH) and two inhibitory patterns (FS and LTS). Although the author provides the specific parameters, we could go further and experiment with a range of values for the parameters that enrich popula- tion neurons’ behavior.
2.3 Neural Networks Architectures
Artificial Neural Networks (ANNs), also known as the second generation of neural networks, get inspiration from the human brain’s structure and function. ANNs are powerful com- putational tools to solve complex pattern recognition, function estimation, or classification problems not amenable to other analytical tools. Over time, ANNs have evolved into more powerful and biologically realistic models leading up to Spiking Neural Networks (SNNs) [24]. In history, we can distinguish three generations of neural networks according to their computational unit.
→ The first generation uses the McCulloch-Pitts neurons as computational units, also re- ferred to as perceptrons. They give rise to various neural network models such as mul- tilayer perceptrons (also called threshold circuits), Hopfield nets, and Boltzmann ma- chines. A characteristic feature of these models is that they can only give digital output.
They are universal models for computing systems with digital input and output; also, every boolean function can be computed by a multilayer perceptron with a single hidden layer [52].
→ The second-generation uses computational units that apply an activation function with a continuous set of possible output values to a weighted sum (or polynomial) of the inputs. Besides, neural nets from the second generation can compute functions with analog input and output. Another feature of this generation is that they support learning algorithms based on gradient descent, such as backpropagation [52].
→ Experiments arrive at the third generation of neural network models that employ spiking neurons as computational units based on neurobiology. Studies of the cortical pyramidal neurons have shown that individual spikes’ timing as a mode of encoding information is essential in biological neural networks. In that sense, encoding information in those neurons is strongly related to the spikes times [24].
The focus of this thesis work is on the third generation of neural networks or SNNs.
However, most of the literature’s implementations are for ANNs, but they could help trace a route toward the SNNs. Currently, today’s von Neumann CPU architectures serve reasonably well at ANNs, and even further, GPU assists its performance optimized for streaming matrix arithmetic. On the other hand, those conventional architectures poorly serve SNN models.
Just as the value of ANNs was not fully appreciated until the advent of sufficiently fast CPUs and GPUs, the same could be the case for spiking models [17]. However, it is vital to know the existing architectures for ANNs to find a way to take positive results to apply them at SNNs.
In this sense, we mention in the next part some crucial general architectures, considering the most used. Literature reports multiple variations for each network, but we try to describe each one’s more general and vital points, emphasizing that there is little or null research in those architectures applied to spiking networks.
2.3.1 Interconnection Architectures
The next major challenge after review the single neuron is implementing a network, specifi- cally the neuron interconnection. Indeed, the net’s full power arrives from considering the en- tire neuron population as a whole. There are several interconnection schemes in the literature for solving specific tasks, but in current days, a universal strategy to solve any computational problem does not exist because of the natural world’s complexity with multiple variables. In this part, we review fundamental architectures in a brief and general snapshot of each one.
→ Feed Forward is a multi-layer neural network in which a group of hidden layers sepa- rates the input and output layers. This architecture has successive layers that feed into one another in the forward direction from input to output. The default architecture of feed-forward networks assumes that all nodes in one layer are connected to those of the next layer [1] (Figure 2.7).
2.3. NEURAL NETWORKS ARCHITECTURES 21
Figure 2.7: Basic Architecture of a Feed Forward Architecture [1].
→ Convolutional neural networks are biologically inspired networks used in computer vision for image classification and object detection. This architecture has historically been the most successful of all types of neural networks. They are used widely for image recognition, object detection/localization, and even text processing [1] (Figure 2.8).
Figure 2.8: LeNet-5: One of the earliest convolutional neural networks [1].
→ Caps Net is a recent approach in which a capsule is a group of neurons whose activity vector represents the instantiation parameters of a specific type of entity such as an object or an object part. As human vision ignores irrelevant details by using a carefully determined sequence of fixation points to ensure that only a tiny fraction of the optic array is ever processed at the highest resolution, Caps Nets try to replicate this behavior [70] (Figure 2.9).
Figure 2.9: A simple CapsNet with 3 layers [70].
2.3.2 Further Details for Multilayer Feedforward ANN
Multilayer feedforward networks are the most popular ANNs, and they are the simplest of neu- ral networks. Although ANNs have essential advantages and exist better and more complex architecture proposals, there are relatively few works for SNNs; thus, feedforward network is the start architecture before exploring another spiking network architectures. Despite its simplicity, this network could open the options to contribute to the massive interconnection problem.
The multilayer feedforward net consists of the first layer called the input layer, the last layer called the output layer, and the layers between the input and output are the hidden layers, as Fig. 2.7 shows. For the formal description of the neurons [81], we can use the so-called mapping function Γ that assigns for each neuron i a subset Γ(i) ⊆ V which consists of all ancestors of the given neuron. Then, a subset Γ−1(i) ⊆ V that consists of all immediate predecessors of the given neuron i, which means in default feedforward net the set of neurons in the previous layer. The connection between the ith and jth neuron is characterised by the weight coefficient wij and the ith neuron have a bias coefficient bi. The weight coefficient reflects the degree of importance of the given connection in the neural network. The output value (activity) of the ith neuron xiis determined by Equations 2.4 and 2.5:
xi = f (ξi) (2.4)
ξi = bi+ �
j∈Γ−1i
wijxj (2.5)
where ξi is the potential of the ith neuron and function f(ξi)is the so-called transfer function. In other words, as Figure 2.10 is shown, a neuron ith with three connected neurons, first, is computed the summation of the potential of all the j neurons in the proportion as the respective weight indicates and it is added the bias constant. Then, an activation function is applied to get the potential of the neuron ith. Although activation function is not restricted, few have been tested to work well in practice as logistic sigmoid, hyperbolic tangent, and rectified linear unit.
Figure 2.10: Representation of a connection for neuron i with 3 connected neurons in a typical feedforward network.
2.4. LEARNING FOR NEURAL NETWORKS 23
2.4 Learning for Neural Networks
Although it is conceptually more straightforward for architectures to create the analogy from ANNs to SNNs as we explore later, the learning process is slightly more complicated because there are significant differences. Even there are no answers to many questions in this regard.
Besides, SNNs incorporate time as an explicit dependency in their computations, making it more potent than ANNs [17], but it makes it more complex to analyze. There is no definitive math demonstration, only clues and our brain, which is the final product of an appropriate interconnection. From biology, even though learning is not transparent, we can identify three principal brain codes that make it functional [40]:
→ Frequency Code: it is the most used since ANNs. The importance is the number of spikes per unit time, which creates a certain proportion between the input and output.
→ Temporal Code: the importance is the sequence of time between each spike, differen- tiate the burst of the regular spikes, and open up the ability to synchronize between neu- rons, which is involved in associative learning through recognition of certain rhythm.
→ Population Codes: the brain can recognize patterns of several neurons to determine an action. In neuroscience is a stimulating discussion of why the brain uses several neurons instead of a small group and has two possible conclusions. More neurons implicate a higher precision, or it is a resistance mechanism to injury because if a single neuron fails, there are a lot of them that continue working.
Also, neurons can exhibit more than one code to entire robust architecture, making it complex to develop a process to leverage spiking neurons’ full potential. Nevertheless, we can simplify and make some considerations to get a helpful training algorithm. In this regard, also, we can find supervised and not supervised algorithms; like we look to solve classification tasks, we focus on supervised learning. Hence, we describe the main techniques for training an SNN, although multiple literature variants improve each one.
→ SpikeProp was created by [9] as a supervised learning rule, akin to traditional error BackPropagation, but to train SNN’s. This method consists of computing the error at the output and distributing backward to hidden layers, and applying gradient descent.
Researchers continuously improve this technique from different aspects, from one layer to multi-layers networks, and consider the first spike until a complete spikes train as [86] to manage several spikes.
→ Genetic Algorithm looks for the weights through the evolutionary process, also could manage any network architecture due to the genetic process does not depend on it but the individuals’ codifications to represent the values to adjust. It is a primary method from the biology approach because it evolves its structure and functionality through both the lifelong learnings of an individual and the evolution of many such individ- uals’ populations. In other words, an individual is a result of the evolution of many generations of populations [42].
