Instituto Tecnol´ogico y de Estudios Superiores de Monterrey

Instituto Tecnol´ogico y de Estudios Superiores de Monterrey

Campus Estado de M´exico

School of Engineering and Sciences

Decoding of motor information from noninvasive

electroencephalographic signals for brain-computer interfaces

A dissertation presented by

Luis Guillermo Hern´andez-Rojas

Submitted to the

School of Engineering and Sciences

in partial fulfillment of the requirements for the degree of Doctor of Philosophy

in

Engineering Science

Atizap´an de Zaragoza, Estado de M´exico January, 2021

Instituto Tecnol´ogico y de Estudios Superiores de Monterrey

Campus Estado de M´exico School of Engineering and Sciences

The committee members, hereby, certify that have read the thesis presented by Luis Guillermo Hern´andez-Rojas and that it is fully adequate in scope and quality as a partial requirement for the degree of Doctor of Philosophy in Engineering Science.

PhD. Javier Mauricio Antelis Ort´ız Tecnol´ogico de Monterrey Principal Advisor

PhD. Omar Mendoza Montoya Tecnol´ogico de Monterrey Co-advisor

PhD. Jessica Cantillo Negrete National Institute of Rehabilitation Committee Member

PhD. Oscar Mart´ınez Mozos Orebro University¨ Committee Member

PhD. Ricardo Caraza Camacho Tecnol´ogico de Monterrey Committee Member

Dr. Jorge Welti Chanes Associate Dean of Graduate Studies School of Engineering and Sciences

Atizap´an de Zaragoza, Estado de M´exico, January, 2021

i

Declaration of Authorship

I, Luis Guillermo Hern´andez-Rojas, declare that this thesis titled, ”Decoding of motor infor- mation from noninvasive electroencephalographic signals for brain-computer interfaces” and the work presented in it are my own. I confirm that:

• This work was done wholly or mainly while in candidature for a research degree at this University.

• Where any part of this thesis has previously been submitted for a degree or any other qualification at this University or any other institution, this has been clearly stated.

• Where I have consulted the published work of others, this is always clearly attributed.

• Where I have quoted from the work of others, the source is always given. With the exception of such quotations, this dissertation is entirely my own work.

• I have acknowledged all main sources of help.

• Where the thesis is based on work done by myself jointly with others, I have made clear exactly what was done by others and what I have contributed myself.

Luis Guillermo Hern´andez-Rojas Atizap´an de Zaragoza, Estado de M´exico, January, 2021

2021 by Luis Guillermo Hern´andez-Rojasc All Rights Reserved

ii

Dedication

I thank God for guiding me along the way and providing me with strength in hard times. I dedicate this work to my parents, whose love, example and effort have taught me that the best things in life are achieved with perseverance, determination and sacrifice. To my lovely wife Andrea, who accompanied me in this exciting adventure, supporting me and encouraging me to follow my dreams. To my brothers and sisters, for making me feel close to Colombia in every video call and encouraging me to continue on my postgraduate studies. They tell me they are proud of my accomplishments,but I am more proud that they are my family.

This thesis is dedicated to them.

iii

Acknowledgements

I would like to express my deepest gratitude to my advisers Dr. Javier Mauricio Antelis Ort´ız and Dr. Omar Mendoza-Montoya for all their support, for spreading my curiosity about the brain and neuroscience and for the invaluable academic mentoring that allowed me to successfully advance in my research.

I am deeply grateful with my friends and colleagues in the Neurotechnology and Brain- Computer Interfaces laboratory at Tecnol´ogico de Monterrey campus Guadalajara. Omar, Efrain, Camilo, John, Alberto, Alejandro and Juan David thank you very much for your in- valuable support. I also want to thank all my teachers and classmates for their hospitality and kindness, they always made me feel at home.

Special thanks to Consejo Nacional de Ciencia y Tecnolog´ıa -CONACYT- and Instituto Tecnol´ogico y de Estudios Superiores de Monterrey Campus Guadalajara for the Ph.D. schol- arship and the financial support provided. This invaluable grant allowed to develop of this thesis.

iv

Decoding of motor information from noninvasive electroencephalographic signals for brain-computer

interfaces by

Luis Guillermo Hern´andez-Rojas Abstract

Brain-Computer Interfaces (BCIs) are emerging assistive technologies that provide an artifi- cial communication pathway between the brain and the external world. These systems trans- late a mental task performed by the user into commands to control external devices using brain signals recorded with invasive or noninvasive techniques. This is remarkably interesting for different applications related with neuromotor rehabilitation field, for example, BCI systems for neurorehabilitation therapies where BCIs provides patients with motor impairments with a non-muscular communication channel that could be used to activate a robot-assisted reha- bilitation device. However, there are other applications not related to the neurorehabilitation field where this technology provides an enhancement for the communication between the user and the its environment. An example of this is the BCI’s for automotive applications, where BCI technology is applied as a part of Advanced Driving Assistance Systems to avoid crash vehicle situations. Irrespective of the type of application, movement-related BCI systems use the motor imagery (MI) paradigm as the mental task that the user performs and which the system detects and classifies by generating commands to drive external devices

Despite the success of Motor Imagery-based BCI systems, there are some characteris- tics of these interfaces that are susceptible to be improved. First, to improve the performance of mental task detection, novel classification models can be explored to compare their perfor- mance with the conventional classification models used in BCI (such as Linear Discriminant Analysis (LDA) and Support Vector Machines (SVM)). Secondly, there are applications in which the motor imagery paradigm has limitations that avoid the BCI system to be able to detect multiple mental motor tasks related to diverse movements generated by the same limb.

In addition, the MI paradigm is not fully adaptable to detect intentions to execute sudden movements, which is important for applications where the objective of BCI is to support and complement the rehabilitation therapies for people with the ability to recover their physical motor functions. Finally, the validation of neurorehabilitation therapies based on BCI online for end users (people with motor disabilities). It is necessary to evaluate the usefulness of this technology in the rehabilitation of patients with motor disabilities.

This PhD thesis investigates the detection of information related to movements from non-invasive EEG signals exploring potential solutions to the limitations of conventional Mo- tor BCI systems. The first study explores novel classification models as those based on deep learning which could improve the BCI system robustness and performance. This study aims to compare classical and Deep Neural Networks (DNN) algorithms for the recognition of Motor Imagery (MI) tasks from electroencephalographic (EEG) signals. The second study

v

investigates the detection of emergency braking from driver’s electroencephalographic (EEG) signals that precede the brake pedal actuation. EEG signals were classified using support vector machines (SVM) and convolutional neural networks (CNN) in order to discriminate between braking intention and normal driving. The third study assess the feasibility of rec- ognizing two rehabilitative right upper-limb movements from pre-movement EEG signals.

These rehabilitative movements were performed self-selected and self-initiated by the users using a motor rehabilitation robotic device. We proposes diverse anticipatory detection sce- narios that discriminate EEG signals corresponding to non-movement state and movement intentions of two same-limb movements. Finally, the last study is focused on the development of a BCI-driven functional electro-stimulation system (FES) aimed at neurorehabilitation of the upper limbs of patients with spinal cord injuries (SCI). Furthermore, clinical benefits of the BCI-FES system in SCI patients are explored by estimating quantitative EEG parameters for motor rehabilitation.

vi

List of Figures

1.1 The design and operation of a brain-computer interface (BCI) system. The BCI system stages are shown inside the yellow box. Firstly, the user’s brain activity is acquired, this signal is digitized in order to extract features through signal processing techniques. The signals are processed to measure features (such as amplitudes or power changes of EEG rhythms) that reflect the user’s intention. These features are translated into commands that operate applica- tion devices [216]. . . 3 2.1 (a) snapshot of the experimental setup and (b) time sequence of a trial during

the execution of experiment. . . 9 2.2 Illustration of the CNN algorithm used in this work, which consisted of two

pairs of convolution and pooling layers followed by a feed forward ANN. . . 11 2.3 (a), (b) and (c) shows a representation of average of all PSD values obtained

participant 3 in the three studied classes: (a) corresponds to relax, (a) to left MIand (a) to right MI. . . 13 2.4 Distribution of classification accuracy achieved with SLP, DNN, DNN+drop

and CNN for each participant (P₁ to P₈)%. The horizontal dotted black line represents the significant chance level or accuracychance. . . 14 3.1 (a) Illustration of the driving system and environment. The driving system

consisted of a commercial set of pedals, steering wheel and gear lever for driv- ing simulators, a 19 inches flat-screen and a computer to control the execution of the experiment. The driving environment consisted of an oval track and contained the participant’s vehicle and a guide vehicle. (b) Image of driving environment in a first person perspective as seen by the participants. (c) EEG electrode locations used in this experiment. (d) Snapshot of the experiment with a participant wearing the EEG electrodes (the participant gave written informed consent to publish this picture). . . 20 3.2 Illustration of the temporal sequence of a morning session (top) and an af-

ternoon session (bottom). Each session consisted of 8 blocks. Each block contained 15 emergency braking of the same experimental combination. The experiment was carried out in four sessions (two in the morning and two in the afternoon). In total, 120 emergency braking situations were presented in each session yielding to a total of 480 emergency braking situations per participant. 22

vii

3.3 Graphical illustration of the data segments and the three types of epochs ex- tracted from each of them: (i) Non-braking epochs: signals of 1.5 s that are more than 3 s apart from any stimulus and/or response (data without emer- gency braking); (ii) pre-response epochs: signals in the time interval [-1.5, 0]

s where the reference t = 0 corresponds to the response (the first notable de- flection of the participant’s vehicle brake pedal); (iii) post-stimulus epochs:

signals in the time interval [0, 1.5] s where the reference t = 0 corresponds to the stimulus (the guide vehicle switching-on of the rear brake lights). . . 24 3.4 Illustration of the CNN algorithm implemented to discriminate between emer-

gency braking intention from normal driving. The CNN consists of two pairs of convolution and pooling layers followed by a feed forward ANN. . . 26 3.5 (a) Distributions of BRT across all participants for each experimental combi-

nation. At least one of the distributions is significantly different which indi- cates different BRT across the experimental combinations. (b) Time-resolved LEG signal averaged across all participants for all experimental combination.

The distribution of BRT across all participants and experimental combination is also shown. The reference time t = 0 correspond to the stimulus. The LEG signal begins to increase at ∼600 ms (i.e., prior to the median BRT) and peaks at ∼850 ms. . . 29 3.6 (a) Distribution of classification accuracy achieved with SVM and CNN for

each participant (P1 to P7). The horizontal dotted red line represents the sig- nificant chance level or accuracy_chance. (b) ROC curves for SVM classifiers for each participant. (c) ROC curves for CNN classifiers for each participant.

The diagonal dotted black line represents the threshold AU C = 0.5 . . . 31 3.7 (a) Distribution of classification accuracy achieved with SVM and CNN for

the classification experiment leave-one-out participant. The horizontal dotted red line represents the significant chance level or accuracychance. (b) ROC curves for SVM classifiers for each leave-one-out participant. (c) ROC curves for CNN classifiers for leave-one-out participant . . . 33 3.8 Illustration of classification accuracies in the train (blue curve) and test (red

curve) sets during training epochs across all participants in the proposed CNN model to to discriminate between emergency braking intention from normal driving. Note that these results shows that the accuracy of both, train and test sets, increases with the number of epochs up to a steady state while no overfitting in the train set or drop in the test set is observed. . . 35 3.9 Illustration of the loss in the train (blue curve) and test (red curve) sets dur-

ing training epochs for across-all-participants in the proposed CNN model to discriminate between emergency braking intention from normal driving. Note that these results shows that the loss of both, train and test sets, decreases with the same tendency throughout epochs. . . 36

viii

3.10 Illustration of classification accuracies in the train (blue curve) and test (red curve) sets during training epochs across leave-one-out participants in the pro- posed CNN model to discriminate between emergency braking intention from normal driving. Note that these results shows that the accuracy of both, train and test sets, increases with the number of epochs up to a steady state while no overfitting in the train set or drop in the test set is observed. . . 37 3.11 Illustration of loss in the train (blue curve) and test (red curve) sets during

training epochs for across leave-one-out participants in the proposed CNN model to to discriminate between emergency braking intention from normal driving. Note that these results shows that the loss of both, train and test sets, decreases throughout epochs up to converge. In addition, note in figures b, c, d, f and g that loss converges quite early for both sets. A similar behavior is shown by the precision-recall curves in figures s5 b and d. This may indicate a fast learning rate when CNNs are applied to our specific dataset. . . 38 3.12 Plots of (a) Precision-Recall curves for SVM classifiers for each participant.

(b) Precision-Recall curves for CNN classifiers for each participant. (c) Precision- Recall curves for SVM classifiers for each leave-one-out participant. (d) Precision-Recall curves for CNN classifiers for each leave-one-out participant. 39 3.13 Accuracies in the train set (red curve) and test set (blue curve) across epochs

using shuffle labels for the participant classification scenario. These results shows that in all participants, the accuracy is fluctuating around the theoretical chance level (50%) and not increment is observed as the epochs increases. . . 40 3.14 Loss in both train (red curve) and test (blue curve) sets using shuffle labels for

the participant classification scenario. These results show a decrease in the loss through the first epochs up to reaching a steady value. This is due to the fact that the CNN model predictions give similar probability values for each of the classes such that the cost function minimizes the loss. The classification model seeks to reduce the loss as much as possible, which is achieved by assigning the same probability to the different classes. This situation has the effect that the classifiers accuracy is at the chance level as shown in figure 3.13. 41 3.15 Accuracies in the train set (red curve) and test set (blue curve) across epochs

using shuffle labels for the leave one out participant classification scenario.

These results shows that in all participants, the accuracy is fluctuating around the theoretical chance level (50%) and not increment is observed as the epochs increases. . . 42 3.16 Loss in both train (red curve) and test (blue curve) sets using shuffle labels

for the leave one out participant classification scenario. These results show a decrease in the loss through the first epochs up to reaching a steady value.

This is due to the fact that the CNN model predictions give similar probability values for each of the classes such that the cost function minimizes the loss.

The classification model seeks to reduce the loss as much as possible, which is achieved by assigning the same probability to the different classes. This situation has the effect that the classifiers accuracy is at the chance level as shown in figure 3.15. . . 43

ix

4.1 (a) Snapshot of the experimental setup. Participants were seated in front of a computer screen while grasping the robotic device with the right arm. The ex- ecution of the experiment was guided by visual cues presented on the screen while the movements were selected and initiated by the participants. (b) Illus- tration of the two movements carried out by the participants. The top image shows the flexion/extension of the right arm while the bottom image shows the supination/pronation of the right forearm. (c) The temporal sequence of a trial. The fist cue indicated to relax. The second cue indicated to execute any of the movements (self-selected) whenever he or she wanted to initiate (self-initiated). The third cue indicated the completion of the task and resting, blinking or moving is allowed if it is necessary. (d) The temporal sequence of an experimental session. The experiment was conducted in two runs, with four blocks in each run, and fifteen trials in each block. This resulted in 120 trials in total per participant. . . 49 4.2 (A) illustration of the movement initiation in different trials and their exclu-

sion procedure using the movement onset time obtained from the digital sig- nals provided by the Tee-R. Trials with movement onset time lower than 1s (early movement initiation) and higher than 11s (delayed movement initia- tion) relative to the presentation of the second visual cue were excluded. (B) illustration of the time aligning procedure in accepted trials. The time axis of each trial was aligned with the movement onset obtained from the digital signals provided by the Tee-R. The duration across all trials is t_end− t_ini = 15s and they comprise a relax or no-movement segment in the time interval [t_ini, t_ini + 3)s (light green segment), a movement execution segment in the time interval [0, t_end)s, and a movement intention segment that precedes the beginning of the movement at t = 0s (light gold segment). In addition, the power spectral density features were computed separately from EEG in the re- lax interval [t_ini+1, t_ini+2)s (dark green segment labeled as Relax) and from EEG in the movement intention interval [−1, 0)s (dark brown segment labeled as IntA or IntB, according to the movement that the participant performed during the trial). . . 52 4.3 Significant event-related desynchronization/synchronization (ERDS) activity

computed across-all-participants in sensors located above the motor cortex.

Abscissa represents time (from −3.1 to 1 s) while ordinate represents fre- quency (from 1 to 40 Hz). Dotted black vertical lines in all graphs represent the movement onset or t = 0 s. Significant desynchronization is presented in blue (i.e., negative percentage values), significant synchronization is pre- sented in red (i.e., positive percentage values), and no significant desynchro- nization/synchronization is presented in green (i.e., zero percentage values).

Significant desynchronization (p < 0.05) is observed in all sensors in the motor-related α [8, 13] Hz and β [14, 30] Hz frequency bands from t ≈ −1 s up to t = 1 s. . . 57

x

4.4 Power spectral density (PSD) and r-squared analysis for participant 14. (a) PSD averaged across-all-trials separately for condition Int ∈ {IntA, IntB}, IntA, IntB and Relax in sensors located above the motor cortex. The PSD roughly around the motor-related α [8, 13] Hz and β [14 − 30] Hz frequency bands is higher in Relax than in the other conditions. (b) r-squared analysis results for Int vs. Relax, IntA vs. IntB and IntA vs. IntB vs Relax. For the three cases, the highest discriminative power is observed roughly around the α frequency band and to a lesser extent in the β frequency band mainly in

sensors located above the motor strip. . . 59

4.5 Across-all-participants distributions of classification accuracy (CA and CA_random) and confusion matrix results obtained in the three offline classification scenar- ios. (a) and (d) correspond to bi-class scenario Int vs. Relax, (b) and (e) correspond to bi-class scenario IntA vs. IntB, and (c) and (f ) correspond to three-class scenario IntA vs. IntB vs. Relax. . . 60

4.6 Across-all-participants rates of time-resolved movement intention detection accuracy, DA(t) (solid blue line) and DA(t)_random (solid orange line), ob- tained in the two pseudo-online classification. DA(t) and DA(t)_random for (a) the bi-class scenario Int vs. Relax, (b) the three-class scenario IntA vs. IntB vs. Relax for supination/pronation of the forearm test trials, and (c) the three-class scenario IntA vs. IntB vs. Relax for flexion/extension of the arm test trials. Green shaded areas show significant differences between DA(t) and DA(t)_random. Vertical dotted lines indicate the movement inten- tion onset or tM I and vertical solid lines (t = 0.00)s represent the movement onset time. . . 62

5.1 Experimental design. . . 70

5.2 BCI system archichecture . . . 71

5.3 (a) training paradigm (b) validation paradigm . . . 75

5.4 Significant event-related desynchronization/synchronization (ERDS) activity computed for HS5 participant session 2 and discriminated by the training or online validation stage and each of the different MI task trials: (a) Left MI trials in training stage, (b) Right MI in training, c) Left MI trials during online validation stage and, (d) Right MI trials during online validation. Abscissa represents time (from 1 to 24 s) for trials during training stage and (from 1 to 10 s) for trial during online validation stage while ordinate represents fre- quency (from 1 to 30 Hz) for all the cases. Significant desynchronization is presented in blue (i.e., negative percentage values), significant synchroniza- tion is presented in red (i.e., positive percentage values), and no significant desynchronization/synchronization is presented in green (i.e., zero percent- age values). . . 82

xi

5.5 Significant event-related desynchronization/synchronization (ERDS) activity computed for SC2 participant session 11 and discriminated by the training or online validation stage and each of the different MI task trials: (a) Left MI trials in training stage, (b) Right MI in training, c) Left MI trials during online validation stage and, (d) Right MI trials during online validation. Abscissa represents time (from 1 to 24 s) for trials during training stage and (from 1 to 12 s) for trial during online validation stage while ordinate represents fre- quency (from 1 to 30 Hz) for all the cases. Significant desynchronization is presented in blue (i.e., negative percentage values), significant synchroniza- tion is presented in red (i.e., positive percentage values), and no significant desynchronization/synchronization is presented in green (i.e., zero percent- age values). . . 83 5.6 (a) BSI index distributions discriminates for Healthy group, SC1 and SC2

participants. the p-values are obtained from Wilcoxon rank-sum tests. (b) and (c) correlations between BSI and classification accuracies using Pearson’s method for Left MI and Right MI training trials, respectively. (d) and (e) correlations between BSI results and FES activation time for Left MI and Right MI online validation trials. The red lines corresponds to the regression line for each correlation analysis. . . 85 5.7 (a) LC distributions per session for HS group, (b) LC distributions per session

for SC1 participant and (c) LC distributions per session for SC2 participant. . 86 5.8 (a) and (b) correlations between LC results in the β_high frequency band and

classification accuracies using Pearson’s method for Left MI and Right MI training trials, respectively. (c) and (d) correlations between LC results in the β_high frequency band and FES activation time for Left MI and Right MI online validation trials. The red lines corresponds to the regression line for each correlation analysis. . . 87 5.9 Correlation plots between β/α ERD/ERS indexes and FES activation time

for online validation trials. (a) shows the correlations found for C4 channel in Left MI trials. (b) shows the correlation results obtained for C3 channel in Right MI trials. The red lines corresponds to the regression line for each correlation analysis. . . 88

xii

List of Tables

2.1 Summary of classification accuracy results for each participant (P₁ to P₇) achieved with the SLP, DNN, DNN+drop and CNN. The lower row shows the grand-average across-all-participants. . . 13 3.1 Description of the eight experimental combinations with absence (–) or pres-

ence (X) of stress, workload and fatigue that were considered during the re- alization of emergency braking. Combination Codoes not include any factor.

Combinations C_s, C_w and C_f only include stress, workload and fatigue, re- spectively. Combinations Cs+w, Cs+f, Cw+f and Cs+w+f include two or three factors. . . 21 3.2 Summary of BRT (units of ms) for the eight experimental combinations. The

average BRT is above 700 ms in experimental combinations that include work- load. . . 28 3.3 Summary of classification accuracy results for each participant (P₁ to P₇)

achieved with the SVM and CNN. The lower row shows the grand-average across-all-participants. . . 32 3.4 Summary of classification accuracy results achieved with SVM and CNN in

the classification experiment leave-one-out participant. The lower row shows the grand-average across-all-participants. . . 32 4.1 Description of the state of the art of studies reporting decoding of motor infor-

mation of same-limb movements from electroencephalographic brain signals.

SCI: spinal cord injury patients; ME: motor execution; MI: motor imagery;

PMI: premovement EEG information; GCs: Gabor coefficients; MRCP: move- ment related cortical potentials; CSP: common spatial patterns; AUC: area under the curve coefficient; FBCSP: filter bank common spatial pattern; AR:

Auto regresive coefficient; RMS: root mean squared; coefficient ELNN: El- mans neural networks; LDA: linear discriminant analysis; sLDA: shrinkage linear discriminant analysis; SDA: sparse discriminant analysis . . . 46 4.2 Summary of the number of trials and movement onset time for each partici-

pant and the average for all of them. These results are only for trials that were kept after the rejection process. . . 56

xiii

4.3 Significant event-related desynchronization/synchronization (ERDS) activity averaged across-all-participants in consecutive 1s-long segments from −3 to 1 s in the motor-related α [8, 13] Hz and β [14, 30] Hz frequency bands for elec- trodes in the motor strip (C4, C2, Cz, C1, and C3). The results are presented as percentage values. . . 58 4.4 Summary of average classification accuracy results (CA and CArandom) achieved

in the three offline classification scenarios. The results are shown for each participant and all of them (Avg). Grey-highlighted results indicate no signif- icant differences between the medians of the CA and CArandom distributions (p > 0.05, Wilcoxon rank-sum test). . . 61 4.5 Summary of classification metrics (N T_D, tM I, DA_min, DA_max, DA_nomov,

DA_intand DAmov) achieved in the two pseudo-online classification scenarios.

sup/pro: supination/pronation of the forearm; f lex/ext: flexion/extension of the arm. . . 63 5.1 Classification accuracies estimated with cross-validation for the three MI stages

(Left MI, Right MI and Rest). The fourth column indicates the model accu- racy (mean value). The last six rows show the mean and standard deviation (std) of the accuracies for healthy group, SC1 and SC2 participant sessions. . 79 5.2 Online classification performance and the average activation time of the FES

routine in each session (FES ONSET) obtained in the evaluation of the BCI- controlled FES system. Results are reported separately for each MI condition trial of FES activation (Left and Right). For the online classification perfor- mance we reported the number of trials in which the participant reached the activation of the FES (succesful trials) and the rate (trial rate) between the suc- cesfull trials and the total trials conducted per session (30 trials). The last six rows show the mean and standard deviation (std) of the accuracies for healthy group, SC1 and SC2 participant sessions. . . 80 5.3 Correlations between frequency-band related event related (de)synchronization

ERD/ERS indexes and the averaged BCI performance for training and online validation stages (classification acuraccy and FES activation time, respec- tively) for C3, Cz and C4 EEG channels. The correlation results are dis- criminated by Left MI and Right MI trials. Grey-highlighted results indicate significant correlations (P < 0.05, Pearson’s correlation). . . 87

xiv

Contents

Abstract v

List of Figures xii

List of Tables xiv

1 Introduction 1

2 A Comparison of Deep Neural Network Algorithms for recognition of EEG mo-

tor imagery signals 7

2.1 Introduction . . . 7

2.2 Dataset Recording and Preparation . . . 8

2.2.1 Data Recording. . . 8

2.2.2 Experiment Design. . . 8

2.2.3 Data Preprocessing. . . 9

2.2.4 Attributes . . . 10

2.2.5 Deep Neural Networks (DNNs). . . 10

2.3 Experiments and Results . . . 12

2.4 Conclusions . . . 14

3 EEG-Based Detection of Braking Intention Under Different Car Driving Condi- tions 16 3.1 Introduction . . . 16

3.2 Materials and Methods . . . 18

3.2.1 Participants . . . 18

3.2.2 Driving system and environment . . . 19

3.2.3 Bio-Electrical signals . . . 19

3.2.4 Description of the experiment . . . 19

3.2.5 Preprocessing . . . 23

3.2.6 Data analysis . . . 23

3.2.7 Detection of emergency braking intention . . . 24

3.3 Results . . . 27

3.3.1 Braking Reaction Time Analysis . . . 27

3.3.2 Detection of emergency braking intention . . . 30

3.4 Discussion and Conclusion . . . 34

3.5 Supplementary Material . . . 35 xv

4 Anticipatory detection of self-paced rehabilitative movements in the same upper

limb from EEG signals 44

4.1 Introduction . . . 44

4.2 Methods & Materials . . . 48

4.2.1 Experimental Design . . . 48

4.2.2 Participants . . . 50

4.2.3 Robot device . . . 50

4.2.4 Data collection . . . 51

4.2.5 Data preparation and pre-processing . . . 51

4.2.6 Event-Related Desynchronization/Synchronization (ERDS) . . . 53

4.2.7 Detection of movement intention . . . 53

4.3 Results . . . 56

4.3.1 Trial rejection and movement onset time . . . 56

4.3.2 ERDS . . . 56

4.3.3 PSD analysis and feature extraction . . . 58

4.3.4 Detection of movement intention . . . 60

4.4 Discussion . . . 64

5 Validation of a brain-computer interface (BCI) system for patients with spinal cord injury 67 5.1 Introduction . . . 67

5.2 Methods and Materials . . . 69

5.2.1 Brain-Computer Interface . . . 69

5.2.2 Participants . . . 74

5.2.3 Experimental design . . . 74

5.2.4 Data analysis . . . 76

5.3 Results . . . 78

5.3.1 Participants . . . 78

5.3.2 Classification model evaluation . . . 78

5.3.3 Online BCI performance . . . 79

5.3.4 ERD/ERS . . . 81

5.3.5 Cuantitative EEG parameters for motor rehabilitation . . . 84

5.4 Discussion . . . 87

6 Conclusion and future work 91

Bibliography 113

xvi

Chapter 1 Introduction

To perform any action that involves movement such as playing the piano, communicating in written or spoken form, driving a car or even kicking a ball, the human brain must care- fully trigger and control a small, discrete muscular activation in a precise sequence in order to generate the movement successfully [7, 154, 96, 171, 162, 61]. Therefore, the neuronal processes responsible the generation and control of movement are considered with one of the most relevant cognitive brain functions since their correct functioning is critical for the in- teraction between the brain and the muscles. [96, 171]. If the communication between the brain and the body is affected by a deterioration of the neuromuscular communication channel due to a motor functional neurological disorder such as stroke, lateral amyotrophic sclerosis, and spinal cord injury, individual may suffer a partial or total loss normal muscular control.

Depending on the progression and complexity of the neurological disorder, these losses may be expressed in the lack of capacity to execute voluntary muscle movements [61, 195, 171].

People affected by a motor disability face a dramatic reduction in quality of life and social interaction [84, 120, 128].

In the last decades, a promissory technology known as Brain-Computer Interface (BCI) have emerged as a alternative to address the limitations of these motor impairments. BCI is an artificial system that bypasses the body’s normal efferent pathways by providing a direct and non-muscular communication channel between the brain and the environment [216, 215, 57].

Consequently, BCIs can help improve the quality of life for patients with motor disabilities by providing them with more independent and autonomous assistive medical technologies and offers researchers a novel research approach to the study of brain functionality. This poten- tial technology has attracted interest for the development of not only motor rehabilitation and recovery applications but also non biomedical applications such as entertainment, communi- cation and remote device control.

The main idea of a BCI system is to measure and decode the brain activity related to user’s intention which is transmitted as control commands to external devices. A BCI design and its main sequential stages is shown in figure 1.1 and describes as follows :

• Signal acquisition. The BCI systems detect the brain activity as electrical and mag- netic changes, mainly. Hence, a wide variety of techniques may be used to measure these changes. Common acquisition techniques employed are: Functional magnetic resonance imaging (fMRI) which measures changes in magnetic activity produce by the neurons [184], functional near infrared spectroscopy (fNIRS) which monitors blood

1

Chapter 1. Introduction 2

flow changes in the brain associated with brain activity [132], Magnetoencephalography (MEG) which recording magnetic fields produced by brain [116], Electrocorticography (ECoG) which registers electrical changes in the brain activity with invasive sensors [121] and Electroencephalography (EEG) technique which records the electrical activ- ity from the scalp with non-invasive electrodes [218, 57, 9]. The EEG recording the most employed technique for the development of BCI caused by EEG equipment is inexpensive, more portable, and comparatively easier to apply than others techniques.

• Signal processing. This stage aim to reduce non-relevant information embedded into the brain signals. These irrelevant data is produced because brains signals are susceptible to different noise sources (e.g. electromyographic and electrocardiographic signals, power line fluctuations for the EEG technique case).

• Feature extraction. The brain activity used in BCIs are characterized by particular characteristics or features as amplitudes and frequencies in the EEG signals. Feature extraction is the process of identifying relevant features and transforming them in a meaningful information that can be interpreted by the BCI system.

• Feature translation. In feature translation, the features generated by the previous step are discriminated, classified according to the user’s intention and transformed as com- mands to operate the applications. In this step, machine learning analysis is performed.

Typically, supervised machine learning models such as Linear Discriminant Analysis, Neural Networks and Support Vector Machines (SVM) are used in this step [127, 130, 208]. The features translation outcomes are used as commands to control an application.

• Applications. They are the external devices that are controlled with the results obtained from the classification model, therefore the communication between the interface and peripheral devices is fundamental for the correct operation of the BCI system. A variety of applications have been successfully used as wheelchairs, prostheses, exoskeletons, communication interfaces.

Finally, the action executed by the application generates feedback in the user who identifies if the intention has been correctly interpreted by the BCI system or if the generated command is erroneous.

A crucial factor about the operation of a BCI system is user’s intention type that BCI interprets as meaningful information [2]. The mental tasks that the user perform to voluntary must modulate changes in brain activity. These tasks are categorized in two major categories:

evoked mental tasks, which requires user’s attention to be focused on the presentation of a set of external sensory stimuli (visual, auditory, tactile, etc.), which triggers a specific and automatic response in the brain activity, such as P300 evoked and steady state visual evoked potentials; and self-generated mental tasks in which the user performs a mental task that does not depend on external stimulation. The changes in brain activity modulated by this type of mental task that the system associates with a command or action to be executed. An example of a user-generated task is the motor imagery (MI) paradigm[151, 78, 177]. In this paradigm, (MI) the user mentally images movement without physical activity over a long time. MI rep- resents the result of access to neuronal process relating to the intention of a movement. This

Chapter 1. Introduction 3

Signal Acquisition

Signal Processing

Feature Extraction

Feature Translation

Applications

Figure 1.1: The design and operation of a brain-computer interface (BCI) system. The BCI system stages are shown inside the yellow box. Firstly, the user’s brain activity is acquired, this signal is digitized in order to extract features through signal processing techniques. The signals are processed to measure features (such as amplitudes or power changes of EEG rhythms) that reflect the user’s intention. These features are translated into commands that operate application devices [216].

induces power changes in frequency bands of EEG signals obtained mainly from the senso- rimotor brain cortex [148, 41, 57, 124]. Finally, an important factor for the development of BCI systems is to identify when the user performs the mental task. This is categorized into two BCI strategy: Synchronous BCI strategy, where the user interacts with the system during time windows. In the synchronous strategy the system guides the user in what mental task must be performed through instructions (auditory, visual or tactile cues). The advantage of synchronous BCI systems is that they focus the execution of the mental task in known time periods which facilitates the detection of the user’s intention, however this reduces the inter- action between the user and the system. In the Asynchronous BCI strategy, the BCI systems are processing and decoding the user’s mental task all the time. The advantage of this strategy is that it provides a more natural interaction between the user and the system (the user con- trols which mental task to perform and when to perform it). However, it has the disadvantage that the processing and decoding of the user’s intention becomes more complex compared to the synchronous strategy. All these configurations, strategies and characteristics must be con- sidered for the development of an optimal BCI system that, according to its purpose, might restore, rehabilitate, complement or improve the interaction between the brain and external peripheral devices.

MI-based BCI systems have been developed primarily for motor control, neuromotor rehabilitation, and non-medical applications. In motor function restoration, BCI systems en- ables communication with wheelchairs, prosthetic robotic arms, or orthotic devices, which offers the ability to support the execution of complete or partial lost movements. For instance, a hand robotic limb prosthesis can be controlled through the BCI to execute the grasping movements of paralyzed hand. Typical applications of BCI systems for the motor function restoration include the control of robotic wheelchairs [94, 27, 45, 48, 222], robotic limb pros- thetic devices [201, 225, 224, 67] and orthotic devices [149, 19, 39].

In motor rehabilitation, the BCI is employed as a neurorehabilitation tool that enables

Chapter 1. Introduction 4

the user to restore the lost motor functions by promoting the recruitment of brain areas re- lated to the areas affected by the impairment and facilitating neuronal plasticity. Typically, the main applications are related with robotic-based or augmented reality rehabilitation pro- grams guided with a BCI system [113, 180, 31]. In addition to the gain in brain plasticity, other remarkable results of these BCI systems are that they increase the user’s engagement in rehabilitation therapy, because users experience a more control (cognitive and physical) of the therapeutic exercises. Applications of BCIs for motor rehabilitation include control of or- thotic devices for stroke therapy [28, 110, 206, 51, 174, 53], augmented reality [95, 104, 209], robot-assisted therapies [135, 6].

There are also studies which have highlighted the importance of EEG-BCI for non- medical devices. The main goal of these systems is enhancing man–machine interaction. This allows to optimize and to enhance human performance and to achieve potentially novel types of skills [18]. The main applications of these systems are BCI-controlled games [3, 80, 64], monitoring performance capability such as monitoring attention, workload and fatigue mental states in daily activities [130, 42, 102, 223, 44], emotion detection [75, 205], Human-machine communication such as BCI-controlled vehicle driving [66, 65, 81, 109]

Regardless of the purpose of the BCI system is motor recovery or rehabilitation (restora- tion of motor function) or the use of non-medical devices, the conventional BCI configuration has the following main characteristics:

• Electroencephalographic (EEG) technique is the most used in the BCI systems.

• The major meaningful information related to movement neural processes in the Brain signals are located in the frequency bands in the range of [8 − 12] Hz and [13 − 30] Hz also known as α and β brain rhythms [156].

• Time and frequency-based features are used to train the classification models.

• Classical supervised machine learning methods are the most used machine learning methods.

• Motor imagery of different parts of the body to identify different mental task.

• Synchronous BCI strategy is the most used.

Despite the success of BCI for movement control in recent years, the main features used for these BCI systems offer a limited solution that constrains their usability in real situations and in daily life activities. For instance, the conventional way to classify such EEG signals is to employ classical supervised classifiers as Linear Discriminant Analysis (LDA), Support Vector Machines (SVM) and their derivations [208]. This has provided satisfactory results in laboratory based settings. However, EEG-based BCIs require to give a step forward towards applications in real and daily live activities, which requires , require more complex algorithms to maximize the decoding accuracy. Consequently, it is necessary to explore novel classifica- tion models as those based on deep learning. Potentially, this could improve robustness and performance.

Another example is related to the MI strategy, MI often offers a limited solution to allow more natural control of the BCI system [122, 20]. During the execution of the MI strategy,

Chapter 1. Introduction 5

there is an inherent delay between imagination carried out by the user and the physical out- put of the application devices. By contrast, in a motor application, it is necessary that robots devices execute the movements while the user is also performing the mental task in order to promote proprioceptive feedback [212]. The delay is due to the need of the BCI technology to instruct the user through a synchronous sequence of strict and highly controlled visual or au- ditory cues about when the imagination should be initiated. In consequence, the movements performed by the motor devices are not perceived to be naturally controlled by the users.

Additionally, the success of a BCI-based applications requires a match between the mental task performed by the user and the movements performed by the robotic devices. This is not the case with the MI strategy since the user usually imagines body movements that are easy to recognize but that are not the same as the movements performed by the robots [28, 211].

Finally, BCI systems need to be validated in more realistic scenarios. Mainly, BCI has been validated in restrictive and controlled environments that may not be suitable for inferring the potential of these systems in life situations and real end-users. Addressing these challenges this PhD thesis investigates the detection of information related to voluntary movements form non-invasive EEG signals, which could provide more intuitive control of movement appli- cations. This thesis is divided into 4 studies that present the work done with different BCI systems designed for movement applications. The main objective of this thesis was to explore different classification models, BCI mental task strategies, BCI experimental designs and BCI systems validation methods in order to find novel solutions to the challenges described above.

More specifically, the objectives of each study are the following:

1. Study 1: A comparison of Deep Neural Network algorithms for recognition of EEG motor imagery signals [68].

This study compares conventional machine learning models used in EEG-based BCI systems with novel Deep Neural Networks models for the recognition of Motor Imagery (MI) tasks. Real EEG signals recorded in a MI-based BCI experiment were used to evaluate the performance of the proposed models in a multiclass classification scenario using frequency-based features. The objective of this work was to explore the potential application of Deep Neural Networks models for the development of BCI systems in daily live activities with real users.

2. Study 2: EEG-Based detection of braking intention under different car driving condi- tions [70].

The anticipatory recognition of braking is essential to prevent traffic accidents. This study investigated the detection of emergency braking from driver’s electroencephalo- graphic (EEG) signals that precede the brake pedal actuation. The objective of this study was twofold: First, to validate the feasibility of incorporating the BCI technology into advanced driver-assistance systems to carry out early detection of emergency braking situations which could be useful to reduce car accidents. Specifically, The proposal BCI was designed to discriminate the user’s intention to perform the lower limb movements required for emergency braking using non-invasive electroencephalographic (EEG) sig- nals. Second, to explore the potential application of Deep Neural Networks models for the proposed BCI system.

Chapter 1. Introduction 6

3. Study 3: Anticipatory detection of self-paced rehabilitative movements in the same upper limb from EEG signals [71].

This study proposes anticipatory detection scenarios that discriminate EEG signals cor- responding to non-movement state and movement intentions of two same-limb move- ments. The objective of this study was twofold: First, to design an asynchronous BCI experiment. the experimental task was conducted by healthy participants and a motor rehabilitation device for the upper limb. The experiment consisted of two self-initiated and self-selected movements of the right upper limb. Second, to explore a different movement-related mental task. Instead to perform Motor Imagery strategy, the experi- ment was conducted in order to explore the feasibility detecting of movement informa- tion form EEG signals recorded that precedes the movement initiation.

4. Study 4: Validation of a brain-computer interface (BCI) system for patients with spinal cord injury.

This study presents the design, implementation and evaluation of a motor imagery (MI)- based BCI developed to control a FES device for SCI patients who are unable to per- form hand grasping movements in both upper limbs. The experiment is performed in a rehabilitation therapy setting.

The main objective of this study was to evaluate of this BCI technology in rehabilitation scenarios. It is essential to determine if BCI patients can operate this particular mind- controlled application and evaluate the possible effect of a FES system on the user’s experience and performance.

The thesis is organized as follows: the scientific details and the specific conclusions of the four contributions are presented in chapters 2, 3, 4, and 5, respectively. The final conclusion and future work of this research is presented in chapter 6.

Chapter 2

A Comparison of Deep Neural Network Algorithms for recognition of EEG motor imagery signals

This chapter aims to compare classical and Deep Neural Networks (DNN) algorithms for the recognition of Motor Imagery (MI) tasks from electroencephalographic (EEG) signals. Four Artificial Neural Networks (ANNs) architectures were implemented and assessed to classify EEG motor imagery signals: (i) Single-Layer Perceptron (SLP), (ii) Fully connected Deep Neural Network (DNN), (iii) Deep Neural Network with Dropout (DNN+dropout) and (iv) Convolutional Neural Network (CNN). Real EEG signals recorded in a MI-based BCI ex- periment were used to evaluate the performance of the proposed algorithms in the classi- fication of three classes (relax, left MI and right MI) using power spectral based features extracted from the EEG signals. The results of a systematic performance evaluation revealed not significant classification accuracies with SLP (averaged of 33.9% ± 0.0%), whereas DNN (59.7% ± 16.3%), DNN+dropout (58.4% ± 14.9%) and CNN (62.1% ± 15.2%) provided sig- nificant classification accuracies above chance level. The highest performances were obtained with DNN and CNN. This study indicates potential application of DNNs for the development of BCI systems in daily live activities with real users.

This manuscript has been published as [68].

2.1 Introduction

Recognition of motor imagery (MI) mental tasks is an essential part of Brain-Computer Inter- faces (BCI) based on non-invasive electroencephalographic (EEG) signals. The conventional way to classify such EEG signals is to employ classical supervised classifiers such as Lin- ear Discriminant Analysis and Support Vector Machines or SVM [208]. This has provided satisfactory results in laboratory based settings. However, EEG-based BCIs require to give a step forward towards applications in real and daily live activities, which requires to detect with higher accuracy the MI mental tasks carried out by the user. To do so, it is necessary to explore novel classification models as those based on deep learning. Potentially, this could improve robustness and performance. Deep Neural Networks are a sort of machine learning

7

Chapter 2. A Comparison of Deep Neural Network Algorithms for recognition of EEG

motor imagery signals 8

algorithms that use multiples computational models with many processing layers to achieve learning representation of data [54]. DNNs have particularly strong power of discrimination and flexibility to represent data through multiple levels of abstraction [92]. Recently, Deep Learning approaches have been applied in BCI studies with satisfactory results [153]. Never- theless, the number of studies in BCI with these algorithms is still reduced [198].

This work evaluates the performance of four ANNs : (i) Single-Layer Perceptron (SLP), (ii) Fully connected Deep Neural Network (DNN), (iii) Deep Neural Network with Dropout (DNN+dropout) and (iv) Convolutional Neural Network (CNN) in a three-class classification scenario using power spectral (PSD) based features extracted from the EEG signals during motor imagery mental tasks and presents a comparison of their performance. The results showed that DNN (59.7% ± 16.3%), DNN+dropout (58.4% ± 14.9%) and CNN (62.1% ± 15.2%) obtained significant classification accuracies above chance level (33.60%). For SLP results, the accuracies were below chance level. This work shows that convolutional Neu- ral Network can be an effective classification model to obtain highest confident classification accuracy in three different movements states of upper limbs. The rest of this chapter is orga- nized as follows. Section 2 describes details about how dataset was recorded and prepared, how attributes are calculated to extract relevant information from the EEG signals, the differ- ent ANNs architectures used and the performance evaluation process. Section 3 describes the results obtained for PSD analysis and the accuracies obtained to a three-class classification scenario.

2.2 Dataset Recording and Preparation

2.2.1 Data Recording.

Eight healthy subjects voluntarily participated in this study. The experiment was conducted in accordance to the Helsinki declaration. All participants were duly informed about the goals of the research. During the execution of the experiment, EEG signals were recorded from 15 scalp locations according to the international 10/20 system (FC3, FC1, FCZ, FC2, FC4, C3, C1, CZ, C2, C4, CP3, CP1, CPZ, CP2 and CP4) using a g.USBamp with active electrodes (g.tec medical engineering GmbH, Austria). The reference and ground electrode were placed over left earlobe and AFZ, respectively. The EEG signals were acquired at a sampling frequency of 256 Hz and not filtering was applied.

2.2.2 Experiment Design.

The experimental task consisted of many trials of imaging the movement of left or right hands.

This was guided by visual cues presented on the screen (see Fig. 2.1a). A trial consisted of three visual cues (see Fig. 2.1b). The first cue was an image with the text ”Relax” and the subjects were instructed to relax the body without performing any voluntary movement (relax phase). The second cue was an image with of an arrow pointing to the left or to the right and instructed to imagine the movement of the corresponding hand during three seconds (movement imagination phase). The last cue was an image with the text ”rest” and indicated to rest, move voluntary or blink during three seconds (rest phase). 280 trials were recorded

Chapter 2. A Comparison of Deep Neural Network Algorithms for recognition of EEG

motor imagery signals 9

per participant, 140 trials for each condition (left or right hand).

(a)

Relax phase 3 s

Movement imagination phase

3 s

Rest phase 3 s

Or

2 ^!"#$

Relax

1^%&"#$

Rest

3^'!"#$

(b)

Figure 2.1: (a) snapshot of the experimental setup and (b) time sequence of a trial during the execution of experiment.

2.2.3 Data Preprocessing.

EEG signals were low-pass filtered at a cutoff frequency of 45 Hz using a 2nd-order Chebychev- type filter and then common average referenced (CAR). Afterwards, EEG signals were seg- mented in trials starting from the first visual cue and up to the second visual cue. For this research the time interval corresponding to third cue (rest phase) were not contemplated.

Therefore, resulted trials had length of six seconds. The zero time reference was aligned with

Chapter 2. A Comparison of Deep Neural Network Algorithms for recognition of EEG

motor imagery signals 10

starting of second visual clue. Thus, the time intervals [-3,0)s and [0,3]s are rest and motor imagery, respectively. Finally, all the conditions were organized according to the experimental condition (relax,left MI,right MI) to construct the dataset.

2.2.4 Attributes

Power Spectral Density (PSD) of the EEG signals were used as feature to discriminate be- tween three classes: Relax, left MI and right MI. The PSD is the Fourier Transform of the autocorrelation function of a signal and estimates how the average power is distributed as a function of frequency [37]. PSD is highly used in BCI due to the high correlation between the MI tasks and the spectral power changes [192]. In the case of brain activities related to MI, the spectral power changes are found over sensory-motor cortex areas in frequencies be- tween [8 - 30]Hz, also known as α and β brain rhythms. For this reason, for each electrode, it was selected the frequency range between 8 and 27 Hz at a steps of 1 Hz to compute the PSD. Thus, the number of the PSD features for each electrode is 20. This yield to the dataset {Xi, ~yi}^N_i=1, where Xi ∈ R^300×1 or Xi ∈ R^15×20, ~yi were labeled as {rest, left MI, right MI}

and N = 560.

2.2.5 Deep Neural Networks (DNNs).

In this research, 4 DNNs architectures were implemented and assessed to classify EEG motor imagery signals: i) single layer perceptron (SLP), ii) fully connected deep neural network (DNN), iii) deep neural network with dropout (DNN+drop) and iv) convolutional neural network (CNN). All of them were implemented with Tensorflow Library [1] and executed in a Geforce GTX Titan Xp GPU (Nvidia,USA).

2.2.5.1 Single layer perceptron (SLP).

The single layer perceptron is the simplest model of neural network and consist of a single neuron with adaptable weights and bias. The importance of SLP lies in its ability to clas- sify patterns there are linearly separable. This characteristic was demonstrated by Rosenblatt [161]. The perceptron model is described as :

y_k = ϕ

p

X

j=1

w_kjx_j − θ_k

!

(2.1)

where w_kj is the k synaptic weight of the neuron, x is attribute j, θ_kis the bias and ϕ(·) is an activation function. For this research, SLP was implemented with 300 adjustable weights and as activation function was used soft-mask unit.

2.2.5.2 Fully connected Deep Neural Network (DNN).

DNN (also known as Multi-layer Perceptron) consists of a three-layer architecture: i) an input layer that functions as an information receiver and has multiple sensory units, ii) one o more hidden layers that makes a non-linear transformation of the input space into a high dimen- sional space and iii) an output layer that gives the network response through an activation

Chapter 2. A Comparison of Deep Neural Network Algorithms for recognition of EEG

motor imagery signals 11

100 feature maps

1^stconvolution layer 1^stMax-pooling layer 2^ndconvolution layer 2^ndmax-pooling layer Feed forward ANN 20

15 15

20

10 8

10

8 4

5 Input layer

50 feature maps

50 feature maps 100 feature maps 2000

output 100

3

Soft-mask function

Figure 2.2: Illustration of the CNN algorithm used in this work, which consisted of two pairs of convolution and pooling layers followed by a feed forward ANN.

function. The aim of DNNs is to solve problems that cannot be separated linearly. This type of network is trained with the back-propagation learning algorithm [15]. In this research was employed a DNN with input layer of 300 nodes, 4 hidden layers, each one with 200, 100, 60, 30 nodes, respectively. For last, soft-mask unit was used as activation function.

2.2.5.3 Deep Neural Network with Dropout (DNN+dropout).

One of the main problems in the previous DNN model is overfitting. To prevent this, it is used a technique called dropout. This technique allows to constrain the amount of nodes in a hidden layer without losing learning performance [191]. To identify which nodes must removed is random (all the dropped nodes has a fixed probability p independent of other units, where p can be chosen using a validation set or can simply be set at 0.5). Thus, the algorithm makes a lot of iterations trying to find the iteration with better accuracy during a validation phase.

In this work, dropout technique was implemented for the DNN model above described with dropout rate was 10% of nodes.

2.2.5.4 Convolutional Neural Network (CNN).

CNN is a sort of supervised deep learning algorithm [92, 54] that have demonstrated notable results in the classification of data with grid-like topology [85]. CNN architecture is composed of: i) a stack of building blocks with convolution kernels and pooling operators and ii) a feed forward Artificial Neural Network (ANN). Each building block extracts relevant information from input map to its own significant feature maps. Finally, the feature maps feed the ANN to compute each class probabilities (using soft-mask function). The equation that describes the convolutional operation is:

S(i, j) = (I × K)(i, j) =X

m

X

n

I(i + m, j + n) · K(m, n) + b (2.2) A building block consists of i) a kernel K of size m×n, which are convoluted (slided over the input map spatially) with input map to construct output feature map S(i, j), ii) an activation function that applied to each convoluted output feature map, iii) a pooling layer that reduces the dimension of the convoluted maps trough operations as average or maximum. In a CNN, the number of building blocks, kernels, kernel’s size, pooling’s size and the structure of the

Chapter 2. A Comparison of Deep Neural Network Algorithms for recognition of EEG

motor imagery signals 12

feed forward ANN are adjustable hyperparameters, while the weights and bias in the kernels and in the feed forward ANN are parameters that are learned from a training set. Learning is typically carried out by the gradient descent method through the backpropagation algorithm [90]. The architecture of the CNN employed in this work is illustrated in figure 2.2. It consists of two pairs of building blocks followed by a feed forward ANN with a hidden layer. The first block consisted of 50 kernels of size 4 × 4, the rectified linear unit as activation function and maximum pooling with non-overlapping regions of size 2 × 2. This resulted in 50 feature maps of size 4 × 5. The second block consisted of K = 100 kernels of size 4 × 4 with the same characteristics of first convolution-pooling block. This resulted in 100 feature maps of size 2 × 3. The feed forward ANN consisted of 600 input neurons, one hidden layer with 100 neurons and 3 neurons in the output layer. The activation function in the hidden layer is the sigmoid while in the output layer is the soft-max.

2.2.5.5 Performance Evaluation.

The total data was splitted in two mutually exclusive sets. The training set consisted of 80%

of the data and the rest 20% of the data as evaluation set. The classifiers are trained using the training set and final classification is performed on the evaluation set. In training, the algorithms were trained in 400 steps. In each step, a batch data is sampled from training data (20% of training data) with which classification model is fed and at final of the training steps the model for evaluation is obtained. Performance metric was classification accuracy which was computed as:

accuracy = T P + T N

T P + T N + F P + F N (2.3)

where T P is the true positive rate, T N is the true negative rate, F P is the false positive rate and F N is the false negative rate. This procedure is repeated 100 times and the distribution and mean ± std of the accuracy metric were computed. The significant classification ac- curacy chance level was the computed with the binomial distribution [34]. The significant classification accuracy chance level is accuracychance = 33.60%. To examine significant dif- ferences between distribution of accuracy and accuracy_chancethe Wilcoxon signed-rank test was applied, while to examine significant differences between three distributions of accuracy the Wilcoxon rank-sum test was applied.

2.3 Experiments and Results

Figure 2.3a, 2.3b and 2.3c shows PSD averaged across all trials for participant 3 in the three studied classes: relax, left MI and right MI respectively. It is observed that in the three classes, the PSD values changes in the range of 8-13 Hz for all electrodes except for FCZ, C1 and C2.

This frequency range corresponds of µ brain rhythms, where decreasing of PSD values are associated with activation of motor imagery processes [152]. To find µ rhythms differences between classes, PSD averaged values in electrodes of different brain locations were exam- ined. For C4, CP4 and CP2 electrodes (placed above right side of brain) PSD averaged values of left MI are smaller than relax and right MI. PSD values decreasing in this electrodes are related to activation of motor imagery processes during a left-side limb movement [105]. For

Chapter 2. A Comparison of Deep Neural Network Algorithms for recognition of EEG

motor imagery signals 13

Frequency (Hz)

8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

Electrode

FC3 FC1 C3 C1 CP3 CP1 CZ FCZ CPZ FC2 FC4 C2 C4 CP2 CP4

0.1 0.2 0.3 0.4 0.5 0.6 0.7

(a)

Frequency (Hz)

8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

Electrode

FC3 FC1 C3 C1 CP3 CP1 CZ FCZ CPZ FC2 FC4 C2 C4 CP2 CP4

0.1 0.2 0.3 0.4 0.5 0.6 0.7

(b)

Frequency (Hz)

8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

Electrode

FC3 FC1 C3 C1 CP3 CP1 CZ FCZ CPZ FC2 FC4 C2 C4 CP2 CP4

0.1 0.2 0.3 0.4 0.5 0.6 0.7

(c)

Figure 2.3: (a), (b) and (c) shows a representation of average of all PSD values obtained participant 3 in the three studied classes: (a) corresponds to relax, (a) to left MI and (a) to right MI.

% Mean ± std

Participant SLP DNN DNN+drop CNN

P₁ 33.9±0.0 62.5±2.1 66.6±2.6 61.3±3.5

P₂ 33.9±0.0 34.4±2.8 34.3±2.6 49.4±3.4

P₃ 33.9±0.0 65.4±2.4 58.3±12.4 85.1±1.8

P₄ 33.9±0.0 64.4±4.9 59.2±11.7 65.4±2.7

P₅ 33.9±0.0 37.7±4.8 41.3±4.7 35.8±1.3

P₆ 33.9±0.0 57.8±6.1 56.0±4.3 56.4±3.4

P₇ 33.9±0.0 80.3±2.6 78.5±2.4 72.0±4.0

P₈ 33.9±0.0 75.0±1.6 72.9±1.8 72.0±4.0

Average 33.9±0.0 59.7±16.3 58.4±14.9 62.1±15.2

Table 2.1: Summary of classification accuracy results for each participant (P₁ to P₇) achieved with the SLP, DNN, DNN+drop and CNN. The lower row shows the grand-average across- all-participants.

C3, CP3, CP1 electrodes (placed above left side of brain) PSD values are smaller in right MI class than relax and classes. This decreasing is associated to right-side limb movement.

The four proposed architectures were evaluated in the three-class classification of relax versus left MI versus right MI. Figure 2.4 shows the distributions of accuracies computed for each participant and under every classification model. For all participants, the median of their dis- tributions of accuracy are higher and significantly different than the accuracy_chance(p < 0.05) except to SLP classification scenarios and DNN and DNN+drop scenarios for participant 2.

Participants 3, 7 and 8 shows the highest distributions of accuracy in range (70-90)% and participants 2 and 3 shows the lowest distributions of accuracy.

Table 2.1 presents the summary of average accuracy for all participants. Also, the mean across all participants in each classification models are presented. Participant 3 in CNN scenario shows the highest mean value (85.1 ± 1.8)%. In the case of SLP, all the partici- pants shows the lowest mean value (33.9 ± 0.0)%. For the average across all participants, CNN shows the highest average with (62.1 ± 15.2)% followed by DNN with (59.7 ± 16.3)%, DNN+drop (58.4 ± 14.9)% and last, SLP with (33.9 ± 0)%.