Desarrollo local: aspectos teóricos fundamentales

Another contribution of this thesis is the incorporation of wearable ultrasonic sensing into the current singly myoelectric sensing based solutions. The feasibility of ultrasonic sensing based dexterous hand motion recognition is first validated among able-bodied subjects. And the drawback of significant sensitivity to probe shift is also identified in the wearable ultrasonic sensing. Along this line, the multimodal fusion based hand motion recognition is investigated using synchronously captured ultrasound and sEMG signals to exploit both the consistency of ultrasound based morphological representation and the relatively better robustness of sEMG to electrode distribution. In particular, the multimodal fusion based pattern recognition approaches are proposed by conducting the feature extraction on both sEMG and ultrasound signals and the subsequent classification of the concatenated feature vectors comprising the myoelectric TDAR features and the ultrasonic linear fitting coefficients (LFC), thereby improving the recognition accuracy with fused sensing modalities. And the evaluation on an amputated subject is provided as a case study on the targeted group of subjects to verify the superiority of multimodal sensing fusion.

1.3.5

Benchmark for sEMG Based Long-term Hand Motion Recogni-

tion

Last but not least, this thesis contributes to the research community of prosthetic control and hand motor function rehabilitation with a benchmark built for long-term evaluation of sEMG based hand motion recognition. The sEMG signals in inclusion are captured from 10 subjects performing 13 hand motions in consecutive 10 days under a standardised training and data acquisition protocol. Because of the standardised signal capturing protocol of the benchmark, it is straightforward to incorporate new subjects and new samples in the future, which allows the mitigation of current limitation of data size. Specifically, the dataset is made up of low-density sEMG signals captured in a prolonged inter-day scenario, which has not been simultaneously addressed by any other public datasets yet.

1.4

Thesis Organisation

The remaining chapters of this thesis are organised as follows.

Chapter 2 reviews the state-of-the-art work on muscular sensing based hand motion recognition with an emphasis on sEMG driven solutions. A comprehensive understanding of the wearable muscular activity sensing techniques and corresponding hand motion recognition algorithms is provided to the readers. The classic and prominent works and the most recent research perspectives are introduced in details. The concluding section finally summarises the progress and the limitations so far, and outlines the future research directions as well.

Chapter 3 first considers the development of conventional pattern recognition approaches of LDA. Discriminant analysis in combination with the subclass division strategy is adopted for long-term sEMG based hand motion recognition across multiple days with the inadequate training data provided. The conventional discriminant analysis is modified by subclass division strategies of unconstrained unsupervised clustering, constrained nearest neighbour based subclass division and implicit subclass division optimisation respectively. Different from the conventional discriminant analysis, subclasses of motion types of multiple days are generated either explicitly or implicitly to accommodate the varying sEMG patterns, which are attributed to physiological changes in long-term use. Explicit subclass division through K- nearest neighbours (KNN) is adopted to generate new labels of the training data by addressing the subclass division and the invariance within sEMG signals of multiple days in multiple subclasses within each motion type. The SDA utilises the subclass division implicitly and seeks the invariance within to reduce or eliminate the burden of re-training. This chapter further demonstrates the feasibility of deep learning approaches in the sEMG based

hand motion recognition, with an emphasis on the low-density electrode distribution based capturing system instead of the commonly used high-density ones. The data segmentation, network structure, pre-training and fine-tuning are routinely introduced in our applications. Specifically, the raw multi-channel sEMG signals are fed to the network for training and classification to further verify the practicality of the deep learning approaches in dealing with both inter-day and inter-subject knowledge transfer for long-term use, which is not considered in the conventional sEMG based hand motion recognition.

Chapter 4 addresses the importance of features from the perspectives of extraction and selection respectively. Both conventional priori knowledge based handcrafted features and deep learning based non-handcrafted features are discussed about and fused into the feature set to achieve a better classification result. A novel feature vector TDARM comprising conventional TDAR features and the TD descriptor enumeration with multiple threshold is proposed and tested for its feasibility. And the evolutionary algorithm BMA is adopted for the feature selection from both classic TDAR features and the ones extracted from multi-length windowed segments to address the need of computational cost reduction and the improvement of recognition accuracy respectively.

Chapter 5 incorporates the ultrasonic sensing modality into the current singly myoelectric modality based muscle activity sensing and hand motion recognition, following the verifica- tion of its feasibility in dexterous hand motion recognition across able-bodied subjects. The LDA classifier in combination with the TDAR features of myoelectric signals and the LFC features of ultrasonic signals is adopted to facilitate the multimodal sensing. The merits of myoelectric and ultrasonic fusion based hand motion recognition are validated with a case study on an amputated subject.

Chapter 6 provides the experiment setup and data acquisition details to form a new sEMG dataset with more subjects and prolonged scenarios involved for long-term use evaluation as a potential benchmark, followed by the experiments and thorough discussion.

Chapter 7 finally summarises the contributions of this thesis and discusses the future research directions.

Literature Review

2.1

Muscle Activity Sensing

The taxonomy of sensing techniques for prosthetic hand control and active motor function rehabilitation is generally described in perspectives of their invasiveness and intuitiveness. Conservative noninvasive modules whose detecting sites are distributed over the skin surface are naturally preferred by prosthetic hand users. Among all the feasible sensing mechanisms, sEMG based myoelectric control has been the most adopted control strategy for decades in both academia and industry. Other noninvasive manifestations like sonomyography (SMG), inertial measurement units (IMU), electrooculography (EOG), electroencephalog- raphy (EEG), mechanomyography (MMG), force myography (FMG) and near-infrared spectroscopy (NIRS) have been utilised independently or in combination with sEMG signals in a multimodal scheme [30, 41, 42]. An invasive sensing modality typically requires surgery process like the electrode implantation or needle insertion for acquisition of intramuscular electromyography (iEMG) [43], the craniotomy and electrode implantation to retrieve elec- trocorticogram (ECoG) [44] and the grafting residual nerves that exert EMG signals to spare muscles through targeted muscle reinnervation (TMR) surgery [45]. Though invasive sensing approaches are rarely exploited as the sensing techniques in a commercial prosthetic hand control system, strategies like TMR surgery are clearly more suitable for proximal amputees, whose muscular structure of arms is no more accessible [6].

To date, let alone the numerous manifestations that represent muscle activity, sEMG remains the main equipped sensing modality for muscular activity sensing in the active control of almost every commercial upper limb prosthesis and exoskeleton for active limb motor function restoration. A main reason is that the EMG signals are directly related to the muscle contraction and provide an intuitive physiological perspective of the motion intention generation. Furthermore, the EMG allows the recognition of motion intention through the

muscle contraction within residual limbs and does not rely on the actual limb movement because of its bio-electric nature. In this thesis, the main focus is attached to the noninvasive sensing techniques with an emphasis on the sEMG driven solutions. This section primarily reviews the myoelectric sensing technique and address the alternative modalities suitable for multimodal fusion based sensing as a complement.