Automatic assessment of motor dexterity for rehabilitation with a representation nonspecific on sensor

(1)

Automatic assessment of

motor dexterity for rehabilitation with a

representation nonspecific on sensor.

Por:

Patrick Heyer Wollenberg

Tesis sometida como requisito parcial para obtener el grado de Maestro

en Ciencias en el Área de Ciencias Computacionales en el Instituto Nacional de Astrofísica, Óptica y Electrónica

Supervisada por:

Dr. Luis Enrique Sucar Succar, INAOE.

Dr. Felipe Orihuela-Espina, INAOE.

Agosto 2016

c

Coordinación de Ciencias Computacionales

INAOE

Luis Enrique Erro 1

Sta. Ma. Tonantzintla,

(2)

(3)

Acknowledgments

Foremost, I would like to express my sincere gratitude to my advisors Dr. Felipe Orihuela-Espina, Dr. Luis Enrique Sucar Succar, for the continuous support of my master study and research, for their motivation, enthusiasm, immense knowledge, and the sometimes needed push. Their guidance helped me in all the time of research and writing of this thesis. I could not have imagined having better advisors and mentors.

I also want to thank my incredible wife Zabdy for tolerating the hours of work, the long hours of support, the attention when I was tired, and the feedback when I was stressed. You are the most incredible wife I could ask for. I love you.

To my sons Leonardo and Oliver, I hope you follow my steps by ﬁnding something that really interests you and allows you to explore the beauty of knowledge. You inspired me to keep going.

Last but not least I extend my greatest gratitude to Dr. Jorge Hernández Franco for his insight in motor dexterity. And also, Dr. Luis R. Castrejón for his incredible help and patients during data capture and for his insight on several of the clinical subjects treated in this work.

THANK YOU ALL!!!

(4)

(5)

List of Tables

2.1. Motor dexterity evaluation . . . 10

2.2. Reduced exercise set . . . 12

3.1. Related work . . . 28

4.1. Variable description . . . 37

4.2. Capture system . . . 37

5.1. Healthy court . . . 49

5.2. Patients court . . . 51

5.3. Average distance and STD between projected data points from diﬀerent sensors for each FMA exercise(Synthetic only data used). This projection distance can be appreciated in Fig 5.7a. . . 57

5.4. Average distance and STD between projected data points from diﬀerent sensors for each FMA exercise (Patient only data used). This projection distance can be appreciated in Fig 5.7b. . . 58

5.5. Different training/classification geometry configurations schemes tested. 60 5.6. Anova for sensors . . . 61

5.7. PSMS selection . . . 63

5.8. PSMS vs Manual . . . 64

5.9. Grand average accuracy . . . 67

5.10. An outline of the features used by the work of [Otten et al., 2015]. . . . 68

(10)

x List of Tables

5.11. Single Factor ANOVA for Naive Bayes, Random Forest, Lineal SVM classiﬁers for all sensing geometry combinations . . . 69

C.1. Average classiﬁcation using both sensors for training and classiﬁcation . 107

C.2. Average classiﬁcation using both sensors for training and only kinect for classiﬁcation . . . 108

C.3. Average classiﬁcation using both sensors for training and only IMU for classiﬁcation . . . 109

(11)

List of Figures

2.1. Upper extremity exercises . . . 11

2.2. Gesture Therapy . . . 13

2.3. 3D Skeleton representation . . . 15

2.4. Illustrative embeddings . . . 17

4.1. Methodology diagram. . . 32

4.2. OpenNI capture . . . 33

4.3. Sensor selection . . . 34

4.4. Sensor conﬁguration . . . 35

4.5. Capture software . . . 36

4.6. Segmentation software . . . 38

4.7. Representation process . . . 39

4.8. Representation process detailed . . . 40

4.9. Representation Separability . . . 41

4.10. Relationship structure . . . 43

4.11. Inheritance structure . . . 44

4.12. FS Inheritance structure . . . 45

5.1. Capture software . . . 49

5.2. Patient capture . . . 50

5.3. Comparative exercises . . . 53

5.4. Angle diﬀerence . . . 55

(12)

xii List of Figures

5.5. Angle diﬀerence tendency . . . 56

5.6. Average diﬀerence trend . . . 56

5.7. Illustrative embeddings . . . 59

5.8. Sensor combination classiﬁcation . . . 62

5.9. PSMS classiﬁcation . . . 64

5.10. IC classiﬁcation . . . 65

5.11. LLC classiﬁcation . . . 66

5.12. SLC classiﬁcation . . . 66

5.13. MLC classiﬁcation . . . 66

5.14. FS-MLC classiﬁcation . . . 66

(13)

Glossary

Fugl-Meyer Assessment Is an clinical instrument to evaluate the motor dexterity of a

patient who suﬀered a CerebroVascular Accidents.

flaccidity A state of complete loss of muscle tone, as occurs in ﬂaccid paralysis of

poliomyelitis.

hemiparesis A general weakness of the contra-lateral side (opposite side of the brain

hemisphere aﬀected), aﬀecting the extremities and face.

hemiparetic Person suﬀering from hemiparesis.

hemiplegia A extreme case of hemiparesis creating total paralysis of the aﬀected side.

motor dexterity assesment Method used by a trained clinician to determine the range

and quality of a patients movement.

spasticity A state of increased tone of a muscle (and an increase in the deep tendon

reﬂexes).

(14)

(15)

Acronyms

2D Two dimensional.

3D Three dimensional.

ADL Activities of Daily Living.

AHRS Attitude and Heading Reference System.

AI Artiﬁcial Intelligence.

ANOVA Analysis of Variance.

BNN Backpropagation Neural Network.

CVA CerebroVascular Accidents.

CVD Cardiovascular Diseases.

DALY Disability Adjusted Life Years.

DALYL Disability Adjusted Life Year Loss.

DCS Dynamic Classiﬁer Selection.

FMA Fugl Meyer Assessment.

FMS Full Model Selection.

(16)

xvi Acronyms

FS Feature Selection.

HGS-PUE Hospital General Sur de Puebla.

HTO-PUE Hospital de Traumatologia y Ortopedia de Puebla.

HUP-BUAP Hospital Universitario de Puebla-Benemérita Universidad Autónoma de

Puebla.

IMU Inertial Measuring Unit.

INAOE Instituto Nacional de Astrofísica, Óptica y Electrónica.

IR Infrared.

M-LC Multi-Label Classiﬁcation.

OT Occupational Therapy.

PA Physical Activity.

PCA Principal Component Analysis.

PSMS Particle Swarm Model Selection.

RGB-D Red Green Blue - Depth.

SDK Software Development Kit.

SVM Support Vector Machine.

t-SNE t-distributed Stochastic Neighbor Embedding.

VGA Video Graphics Array.

WHO World Health Organization.

(17)

Chapter 1

Introduction

Patient evaluation causes a great burden on medics and clinical staff. Given the increase inCardiovascular Diseases (CVD), including stroke and traumatic injuries, this work presents a novel automatized solution to reduce the burden of patient assessment using computational tools such as representations, model selection, classifiers and multi-label classifiers.

1.1. Motivation

As life expectancy increases worldwide, the health problems associated with aging such asCerebroVascular Accidents (CVA) also increase, creating a necessity for health systems to cope with a higher number ofCVD patients (an estimate of 6.7 million world-wide in 2012) requiring treatment. This increment of patients generates a burden for therapists, and health care systems demanding innovative solutions to alleviate this burden. According to [World-Health-Organization, 2014b] "CVD’s are the number one cause of death globally: more people die annually from CVD’s than from any other cause", and as [Yusuf et al., 2001] indicate the estimated burden attributable to CVD’s is 30.9% of deaths in 1998, and a 10.3% of disease related burden, in terms of Disability Adjusted Life Year Loss (DALYL). "Over three quarters of CVD deaths take place in low and middle income countries" [World-Health-Organization, 2014a].

The treatment for these problems, given that there is no known cure, consists of rehabilitation programs designed to restore, compensate or sustitute the function to afford the patient as much independence in daily living as possible. These therapies can focus on many different areas affecting daily living. In this work we focus on therapies

(18)

2 1.2. RESEARCH PROBLEM

dedicated to the treatment of motor disabilities. These therapies generally consist in repeatedly performing exercises or daily living tasks in the expectancy of mapping these behaviors to an unaﬀected part of the brain or recovery of function in the penumbra [Krakauer et al., 2012] . As part of these therapies, evaluation sessions are performed initially to determine the state of the patient and to monitor his progress as therapy advances. These evaluation sessions can take from ten to 45 minutes, depending on the clinical instrument used for the assessment during which an expert therapist is required. This creates an additional burden for therapists.

The main justification behind this work is the need for a reduction of experts burden given the limited number of experts for assessing motor dexterity and the growing number of cases of stroke patients with subsequent cost reduction. Based on this direct necessity, it is considered that automatized assessment is a feasible possibility to improve rehabilitation success by permitting a wider range of patients to benefit from a correct assessment in a timely manner as well as enabling alternatives such as virtual therapy and telerehabilitation. Even if this problem has alredy been the focus of several investigations, the clinical consensus seems to be that the proposed solutionts do not fulfill their standards, such as accuracy and unobtrusive sensing. In this work, we will focus onFugl Meyer Assessment (FMA) [Fugl Meyer et al., 1975], a clinical instrument widely used in clinical and research centers. A secondary opportunity is to improve therapy success as the enhancement of feedback on patients performance gives them on the fly information of their progress during rehabilitation.

1.2. Research problem

Rehabilitation wards generally has some sensing technologies infrastructures in place, but this vary greatly from one center to another, depending on the direct focus of the center and budget. Any solution for automatic assessment has to be able to cope with a certain variety of sensors while being able to capture all the movements presented by Fugl-Meyer Assessment. This assessment is generally used at these settings to evaluate

(19)

CHAPTER 1. INTRODUCTION 3

patients using a series of exercises that encompass many diﬀerent movements.

Diﬀerent clinical evaluation instruments consider diﬀerent relationships among exer-cises, hence the evaluation or assessment of a certain exercise may help to the outcome of other exercise assessments.

The computational problems that will be attended in this proposal are:

1. The proposal of a mathematical representation of human motion for the upper extremities that is independent from the sensor (constrained to a range of sensors) and that oﬀers high class separability.

2. The automatic evaluation of the motor assessment tool, i.e. classification of the curricula of exercises that are part of the Fugl-Meyer assessment from human kinematics represented in the proposed way.

3. The elaboration of a multi-label classification strategy that exploits information from preclassiﬁed classes to enhance classiﬁcation performance.

1.3. Research questions

The following questions arise.

• Provided an intermediate representation of body posture and gestures (sequence of postures); for each element of the FMA test, is there a projection with high predictive power as characterized by some acceptable trade-oﬀ among type I and II errors?

• Is it possible to obtain the information for this representation from a range of low-cost sensing options?

• Does the use of a unique representation for diﬀerent sensor types permit to automatization of the assessment of motor dexterity to a satisfactory level of accuracy surpassing literature reported results?

(20)

4 1.4. HYPOTHESIS

• Does exploitation of multi-label classification strategy improve the classification rate of exercise-wise classification by tapping the information shared among exercises?

1.4. Hypothesis

The hypotheses of this work are:

• The assessment of motor dexterity following a validated clinical test (like Fugl-Meyer Assessment) can be automatically mimicked to (statistically) match the assessment outcomes by experts physiotherapists.

• The classification of gestures differentiated only from the motor dexterity of the performer using a representation nonspecific on sensor, and multi-label classifiers yields results above literature reported rates.

1.5. Objectives

The main goal of this work is to establish a representation of arm kinematics that can be retrieved from a range of sensors, together with a classiﬁcation model capable of determining human motor dexterity from a clinically accepted scale such that the predictive model matches expert outcome when using this scale (while reducing retest variability and permitting ﬂexible sensor conjugation for wide applicability).

1.5.1. Specific objectives

The speciﬁc objectives of this work are:

1. Create a common representation of human upper extremity movement that can be eﬀectively obtained from a range of sensors, thus relaxing the data acquisition setup.

(21)

2. Optimize a classification model for evaluating the quality of gesture made during the execution of exercises present in the Fugl-Meyer test of motor dexterity, exploring different classification methods on the features represented by our model.

3. Verify the implementation with healthy participants introducing controlled errors.

4. Validate the solution on stroke patients against expert assessments of motor dexterity using data from diﬀerent sensors.

1.6. Rationale of the solution

When a movement is captured using different sensors, the underlying information of the movement has to be encoded in some manner to be analyzed. Given that the movement is not dependent of the sensing technology or geometry this information should be susceptible to be represented independently from the sensor used during capture. Consequently, considering that the underlying information from different movements is different, then the resulting representation of the movements should be different in some manner. A representation that is capable of maintaining these differences can be used to identify the movement that generated any particular representation. To fully exploit these properties of a representation it is convenient to enlarge the differences of different movements while maintaining similar movement representations similar. To obtain a high separation of these representations it is possible to project the data from the sensors to a feature space in such a way that these properties can be used by a classification algorithm to differentiate movements. Any projection that increases the differences of dissimilar movements and reduces the differences of similar movements makes it easier to identify the movement that generated the representation.

If a representation has a high separability between dissimilar movements this can be capitalized upon by a classifier to separate different classes. Depending on the way the representation separates different elements it is necessary to look for the classifier that best exploits the data distribution. Given the number of possible configurations a

(22)

6 1.7. PUBLICATIONS

representation has, it is convenient to use some kind of optimization to determine the best classiﬁer for a given dataset.

Since the motor dexterity of a person is not limited to some speciﬁc movement, but rather a group of movements, it should be possible to use known information of one movement to obtain more information of another movement. Under these premises it should be possible to exploit the relationship among diﬀerent movements.

1.6.1. Contributions

As a ﬁnal result of this work we obtained a system that allows the automatic assessment of upper extremity motor dexterity using a clinically validated scale with results surpassing results reported in literature. The main contributions of this work are:

1. A sensor independent representation of human upper extremity movement: con-sisting of a composition of functions that projects the data obtained from a range of sensors to a common space were dexterity class separability is facilitated.

2. A hierarchical multi-label classification algorithm for exploiting inter-related move-ments: that exploits information shared between different exercises by passing information among classifiers using a hierarchical structure, improving classification of individual exercises thanks to the related nature of the FMA exercises.

3. An algorithm to perform automatic FMA capitalizing the proposed representation and classiﬁcation with results statistically similar to expert opinion.

1.7. Publications

The following publications have been a direct result of this research:

• Heyer P. et al Sensor Adequacy and Arm Movement Encoding for Automatic Assessment of Motor Dexterity for Virtual Rehabilitation. 9th World Congress for Neuro Rehabilitation. Accepted 9-13/04/2016

(23)

• Heyer P. et al Sensor Abstracted Extremity Representation for Automatic Fugl-Meyer Assessment. EAI International Conference on Future Internet e-Health. Accepted 25-28/05/2016

• Heyer P. et al Ancestor augmented network-based chain multi-label classiﬁers and their application to the automatization of motor dexterity assessment. In preparation.

1.8. Scope and limitations

The work presented is dedicated to the assessment of motor dexterity of the upper extremity ofhemiparetic stroke patients, in an age range of 35-95 years, being assessed

using FMA scale undergoing Occupational Therapy (OT). The reliability of the proposed methods depends on the correct performance of exercises as deﬁned in Appendix B and the correct placement of the sensors as described in Chapter 4. Any use for the proposed methods other then those explicitly described in this work requires a separate validation study.

This work is limited to sensors that capture human movement in a way that it can be transformed to an limb segment based orientation space. Even though an extension to lower extremities is possible in theory, this work limits to Fugl-Meyer exercises of the upper extremities that do not require external intervention.

1.9. Summary of the proposed methodology and

results

We propose a method of representing data collected during clinical assessment in a sensor abstracted manner, that has a high separability among different motor dexterity levels given the salient component projection proposed. This representation will be used to automatically asses motor dexterity using different classification techniques including

(24)

8 1.10. SUMMARY OF THE THESIS

classification algorithms mentioned in literature. In addition, an optimized classification scheme was selected using full model selection algorithms. Finally a new method of multi-label classification is proposed to exploit information shared among the exercises performed during clinical assessment.

For this, data from 20 patients and 15 healthy participants was collected using different sensing technologies simultaneously. The data capture for the patients were performed at four different clinical settings under medical supervision. The data from healthy participants was collected for classifier training and evaluation purposes.

Using the proposed representation combined with our proposed multi-label classifica-tion scheme an average classificaclassifica-tion accuracy of 93.65±2.75 was obtained compared to

the maximum reported in literature of 93.1±4.0 for healthy subjects. When comparing

our the proposed method whit the methods presented in literature the diﬀerences be-tween averages are even more signiﬁcant, with an average accuracy of 93.65±2.75 from

the proposed method compared to 86.1±5.1 for healthy participants and 89.72±4.43

vs 71.12±5.38 for patients.

1.10. Summary of the thesis

The rest of this work is composed by six chapters that are described as follows. In chapter

2 the theoretical underpinnings that support this work are given. In chapter 3diﬀerent investigations that are related to this work are described. In chapter4 the components of the proposed solutions are described. In chapter 5ﬁndings are summarized and the most relevant results of this work are presented. In chapter6 the conclusions from the work are presented along with possible extensions for this work to be explored in future work. Finally, the main document is accompanied by a series of appendices that include relevant information.

(25)

Chapter 2

Background

This chapter gives a brief introduction to the medical problem of stroke in section2.1, information above rehabilitation in section 2.2. As for sections 2.3, 2.4, 2.5, and 2.6

they focus on the necessary computational tools of representation, model selection, and multi-label classiﬁcation.

2.1. Stroke and motor evaluation

Ictus commonly known as stroke is a kind of CVA were neural oxygenation is disrupted by the interruption of blood ﬂow to a region of the brain. This interruption can be caused by two reasons either blood vessel rupture (hemorrhagic ictus) or by blood vessel clogging (ischemic ictus). CVA are one of the main cause of death and disability worldwide according to [World-Health-Organization, 2014a]. When, following stroke damage occurs on motor neurons, theCVA may causehemiparesis or in extreme cases

hemiplegia, causing a lose of voluntary movement of the extremities on the aﬀected side

and a reduction of resistance to passive movement.

In cases were the patient survive the CVA, the sensor-motor disruption mitigates over the ﬁrst few days, while maximal recovery occurs during the ﬁrst six months. This greatly depends on the localization and the size of the lesion [Krakauer et al., 2012]. During recoveryflaccidity mainly turn intospasticity.

Motor dexterity assessment is the method a therapist uses to evaluate the extension of sensor-motor damage and to measure recovery. This assessment is generally the starting point for rehabilitation therapy, being repeated multiple times during the rehabilitation process or to evaluate patients improvement, to adjust the exercises to his condition. A

(26)

10 2.1. STROKE AND MOTOR EVALUATION

bibliographical review reveals a large number of diﬀerent clinical instruments to perform motor dexterity assessment. In Table2.1 the most common motor dexterity assessment instruments with a general description are summarized.

Scale Acronym Description.

Modiﬁed Ashworth Scale [Bohannon and Smith, 1987]

MAS Measures spasticity in patients with lesions of the Central Nervous System.

Barthel Index [Wade and Collin, 1988]

BI Assesses the ability of an individual with a neuromuscular or musculoskeletal disorder to care for him/herself.

Box and Block Test [Desrosiers et al., 1994]

BBT Assesses unilateral gross manual dexterity. Chedoke-McMaster Stroke

Assessment Measure [ Gow-land et al., 1993]

CMMSAM Assesses physical impairment and disability in clients with stroke and other neurological impairment.

Modiﬁed Rankin Handicap Scale [Banks and Marotta, 2007]

MRHS Categorizes level of functional indepen dence with reference to pre-stroke activities.

Frenchay Activities Index [Schuling et al., 1993]

FAI Assesses a broad range of activities of daily living in patients recovering from stroke.

Modiﬁed Motor Assessment Scale [Loewen and Ander-son, 1988]

MMAS Assesses everyday motor function in stroke patients.

Wolf Motor Function Test [Wolf et al., 2001]

WMFT Quantitative measure of upper extremity motor ability through timed and functional tasks.

London Handicap Scale [Harwood et al., 1994]

LHS Measures health status in patients with chronic, multiple, or progressive diseases, including evaluation of interventions de ployed in their treatment.

Action Research Arm Test [McDonnell, 2008]

ARAT Assesses upper limb functioning using observational methods. Arm Motor Ability Test

[Kopp et al., 1997]

AMAT To evaluate disabilities in upper extremity function in Activi-ties of Daily Living (ADL)using a quantitative and qualitative measure.

Fugl-Meyer Assessment [Fugl Meyer et al., 1975]

FMA Evaluates and measures recovery in post stroke hemiparetic patients.

(27)

(28)

12 2.2. REHABILITATION

Exercise Simplified Instrucctions

Flexor Synergy Move your hand from the opposite knee and to same side ear pointing your elbow outwards.

Extensor Synergy Move your hand from a lateral hanging position to your oppo-site knee without moving your trunk.

Combined Synergy Move your hand from your knee to your lower back.

Combined Synergy Move your hand from a lateral hanging position upwards to 90 degree (pointing to the horizon) without ﬂexing your elbow. PronationSupination With your elbow touching your side, and your forearm pointing

forward rotate your hand (palm up->palm down)

Shoulder Abduction 90 Move your hand from a lateral hanging position sidways to 90 degree.

Shoulder Flexion Move your hand from pointing at the horizon upwards 180 degree (reach up).

Coordination/Speed Move your ﬁnger from the opposite side knee to nose your nose (5 repetitions as fast as possible)

Table 2.2.: Subset of movements performed during FMA of upper extremities with a simplified explanation.

2.2. Rehabilitation

Rehabilitation therapy is the clinical treatment to recover lost abilities due to an impairment [Kim et al., 2016]. In the context of post-stroke rehabilitation there are several different types of therapy each intended to regain autonomy in the daily activities affected by the accident; these therapies can be combined in cases were multiple skills have been affected. In cases of hemiparetic patients the most common therapy used is Occupational Therapy (OT). [Steultjens et al., 2003] found a small but significant effect size for the efficacy of comprehensiveOT. Occupational therapy OTfocuses on daily living activities, in a way that the exercises performed during therapy help the

(29)

CHAPTER 2. BACKGROUND 13

patient to practice skills necessary for their independence during daily routine. The kind of exercises performed during occupational therapy rehabilitation consider repetition of movements to perform reaching, grasping, lifting. Under normal conditions the rehabilitation iterates through the following steps: 1) Patient motor dexterity evaluation, 2) Therapy session design. 3)Exerciser sessions between 30 and 60 minutes depending on patient.

2.2.1. Virtual and robotic rehabilitation

In recent years occupational therapy has branched to submodalities including virtual rehabilitation were the therapeutic exercises are presented by virtual reality technologies used in serious games as presented [Burke et al., 2009; Rego et al., 2010; Sucar et al., 2009]. This paradigm uses the technology used in video games to allow therapy to be performed trough virtual scenarios reducing the implementation and operator costs and further allowing therapy to be performed at a location more comfortable for the patient. One of such systems illustrated in Fig: 2.2is Gesture Therapy [Sucar et al., 2014b], a vision based movement tracking system that uses a patented gripper to interact with a series of games.

Figure 2.2.:Gesture Therapy is a serious game virtual rehabilitation platform that adjusts to patient progress using probabilistic artificial intelligence.

(30)

14 2.3. REPRESENTATION

These kind of systems have a number of benefits compared to traditional occupational therapy, such as; a total control of the rehabilitation environment, and the possibility to simulate scenarios that could be dangerous for the patient in real life (given the patients current limitations) [Rizzo, 2002;Schultheis and Rizzo, 2001] among others. Feasibility studies [Joo et al., 2010] indicate that the arise of new 3D tracking technology is a possible path to improve virtual therapy by using complete body tracking of stroke patients to interact with virtual environments. A common companion of virtual rehabilitation are robotic assisted therapies [Kahn et al., 2006;Prange et al., 2006;Reinkensmeyer et al., 2002; Volpe et al., 2000]. These robots extend the possibilities of virtual rehabilitation by providing different kinds of therapy: passive, active assisted movement, and active-resisted movement. A limitation of virtual rehabilitation and robotic rehabilitation is that they still require trained therapists to define the therapy and to evaluate patients progress which reduces their usefulness for home based rehabilitation, a situation that is changing quickly with the incorporation of artificial intelligence to guide the decision making process [Orihuela-Espina and Sucar, 2016; Sucar et al., 2016]. An example of these advances can be seen in the project Gesture Therapy, were the artificial intelligence guides game adaptation to adjust therapy to patients.

2.3. Representation

The correct performance of an automatized motor dexterity assessment system, is highly dependent of how the information of the movements is represented. A good

representation of the underlying data allows an AI system to retrieve the necessary

information, that is valuable for the decision making process.

One of the most interesting ways to describe this is: "Good representations bring close to each other examples which share abstract characteristics that are relevant factors of variation of the data distribution" [Bengio, 2009]. From a classiﬁcation point of view [Bengio et al., 2013] also states that, "AnArtiﬁcial Intelligence (AI)must fundamentally understand the world around us, and this can only be achieved if it can learn to identify

(31)

(32)

sentation can come from a variety of sensors but generally it is obtained usingInertial Measuring Unit (IMU)’s or inferred from visual information of a single or multi camera setup. This approach works as long as the data is captured from a speciﬁc sensor and adjusted to the 3D skeleton. For a representation that does not depend on the sensor it is important to extract those inherent features of higher variance. This can be achieved by using salient component projection discussed below.

The 3D skeleton representation can generally be improved for classiﬁcation purposes by using the features of higher variance, that means those features that represent as much of the variability in the data as possible. To obtain these features a number of diﬀerent projection functions can be used such asPrincipal Component Analysis (PCA)

[Hotelling, 1933],t-distributed Stochastic Neighbor Embedding (t-SNE)[Van der Maaten and Hinton, 2008] or Isomap [Tenenbaum et al., 2000; Balasubramanian and Schwartz, 2002] among others. For a more complete review of the topic, the reader is directed to [Carreira-Perpinán, 1997]. These projection functions embed high dimensional data to a salient component space, by projecting to a different space were relevant features are preserved, generally reducing dimensionality as a side effect. The effect of some of these projections on a given dataset can be appreciated in Fig2.4. showing some clear advantages for the selection of the adequate projection method on a given dataset.

(33)

(a) Subset of the digits benchmark dataset.

(b)Digits dataset projected to 2D using isomap.

(c) Digits dataset projected to 2D using PCA.

(d)Digits dataset projected to 2D using t-SNE.

Figure 2.4.: Various embeddings of the digits dataset(Images taken from: Manifold learn-ing on handwritten digits: Locally Linear Embeddlearn-ing [Pedregosa et al., 2011]), illustrating how the choice affects the separability of classes.

2.3.1.1. Principal Component Analysis

PCA is a multivariate technique developed by [Hotelling, 1933] that analyzes a set of observations that are described by several inter-correlated dependent variables. Its goal is to extract the important information from the observations, so to represent it as a set of new orthogonal variables called principal components (strictly a rotation of axes that align to maximum variance), projecting these new variables to a map were dimensionality reduction occurs eﬀortlessly by simply removing those components encoding less variance. The projection itself minimizes the sum of Kullback-Leibler

(34)

divergences over all data-points using gradient descent.

The locations of the points yi4 in the map are determined by minimizing the

Kull-back–Leibler divergence of the distributionQ from the distributionP, that is:

KL(P||Q) =X i6=j

pijlog

pij

qij

(2.1)

Mathematically, the transformation is deﬁned by a set of p-dimensional vectors of weights or loadings w(k) = (w1, ..., wp)(k) that map each row vector x(i) of X to a new

vector of principal component scores t(i) = (t1, ..., tk)(i), given by equation 2.2 in such

a way that the individual variables oft considered over the data set successively inherit

the maximum possible variance fromx, with each loading vector w constrained to be a

unit vector.

tk(i)=x(i)·w(k) (2.2)

The ﬁrst loading vector w(1) thus has to satisfy

w(1) = arg max

kwk=1 (

X

i

(t1)2₍_i₎

)

= arg max

kwk=1 (

X

i

x(i)·w

2 )

(2.3)

Since w(1) has been deﬁned to be a unit vector, it equivalently also satisﬁes

w(1) = arg max

(

wT_XT_Xw

wTw )

(2.4)

A standard result for a symmetric matrix such as XT X is that the quotient’s maximum possible value is the largest eigenvalue of the matrix, which occurs whenw is

the corresponding eigenvector.

With w(1) found, the ﬁrst component of a data vector x(i) can then be given as a

scoret1(i) =x(i)·w(1) in the transformed co-ordinates, or as the corresponding vector in the original variables.

(35)

2.3.1.2. t-distributed Stochastic Neighborhood Embedding

t-SNE[Van der Maaten and Hinton, 2008] is a visualization algorithm that embeds data in low dimensional space. For this, it converts similarities between data points to joint probabilities representing similarities among classes.

Given a set of N high-dimensional objects x1, . . . ,xN, t-SNE ﬁrst computes

prob-abilities pij that are proportional to the similarity of objects xi and xj, as follows:

pj|i =

exp(−kxi−xjk2/2σi2) P

k6=ℓexp(−kxℓ−xkk2/2σi2)

, pij =

pj|i +pi|j

2N (2.5)

The bandwidth of the Gaussian kernels σi, is set in such a way that the perplexity

(probability that a given model predicts a sample) of the conditional distribution equals a predeﬁned perplexity using a binary search. As a result, the bandwidth is adapted to the density of the data: smaller values of σi are used in denser parts of the data space.

t-SNE aims to learn a d−dimensional mapy1, . . . ,yN (withyi ∈Rd) that reﬂects the

similaritiespij as well as possible. To this end, it measures similaritiesqij between two

points in the mapyi andyj, using a very similar approach. Speciﬁcally,qij is deﬁned as:

qij =

(1 +kyi−yjk2)−1 P

k6=i(1 +kyk−yik2)−1

(2.6)

Herein a heavy-tailed Student-t distribution is used to measure similarities between low-dimensional points in order to allow dissimilar objects to be modeled far apart in the map.

The minimization of the Kullback–Leibler divergence with respect to the points yi is

performed using gradient descent. The result of this optimization is a map that reﬂects the similarities between the high-dimensional inputs well.

(36)

20 2.4. CLASSIFIER

2.4. Classifier

A classiﬁer is a mathematical function that maps an input to a certain category or class. These functions are generally learned from a group of samples called training set and later, the learned function is used to determine the class of a new element or group of elements called test set.

2.5. Multi-label classification

Multi-label classification has received increasing attention in recent years as several important problems have arised such as: text classification (assigning a document to several topics), HIV drug selection (determining the optimal set of drugs), and scene classification, among others. These problems require the prediction of a set of multiple labels. This is different from traditional (one-dimensional) classifiers as multi-label classifiers assign each sample to N different classes. For the problem of multi-label classification two general solutions have been proposed [Tsoumakas and Katakis, 2006] a) Binary relevance, a method were a series of multiple classifiers is trained for each label set, obtaining a series of labels that combined form the resulting class vector. b) Label power set, in this method the problem is transformed from a multi-label problem into a single-label problem by creating a synthetic output class that is a combination of all possible class combination of the original multi-label problem. These proposals have a series of drawbacks that make them either, overly complex computationally as in label power set(b) or unable to use mutual class information in the case of binary relevance(a). To prevent the problems of the mentioned solutions, chained classifiers have been introduced as a third alternative [Sucar et al., 2014a] ideally getting the best of both worlds. This solution uses a series of classifiers maintaining the simplicity of Binary relevance method, but uses labels of previously classifiers as features for the subsequent classifiers to take advantage of the mutual information shared among classes.

(37)

N class labels [Blockeel et al., 2006; Vens et al., 2008; Zhang and Zhang, 2010]. These classes and their corresponding labels may or may not be related among them, and if they are, class dependencies can be exploded to improve the predictive power of the classiﬁer.

These intermediate approaches vary in the way they pass resolved information to late resolved classifiers, from the most simple list-like chain [Read et al., 2011], to sophisticated hierarchical or tree-like [Vens et al., 2008; Ramírez-Corona et al., 2014] and network-based chain [Rousu et al., 2006; Sucar et al., 2014a] strategies. The chained classifiers still have a grate pitfall to overcome, that is caused by the fact that classifier order can directly affect the resulting outcome.Beyond the scope of this work is a detailed review of different strategies available in the literature to compensate for this shortcoming, but the interested reader can find more information in [Tsoumakas and Katakis, 2006]. However, of interest here is their limitation of dealing with more sophisticated relations among classes. A particular subtype ofM-LCproblems are those in which the classes relate among themselves according to a graph or network for which early list-like chain classifiers might be insufficient. Literature has already suggested more aggressive multi-label classifiers [Rousu et al., 2006; Vens et al., 2008] capable of accommodating such network-based class structures. These network-like structures among classes are often encoded in directed acyclic graphs e.g. [Sucar et al., 2014a]. including the particular case of trees [Vens et al., 2008], were the graph structure is either learned or assumed from prior domain knowledge.

2.6. Model selection

As the number of classification algorithms grows selecting an appropriated algorithm for a given dataset becomes a more difficult task, even more with the even larger combination of possible parameters for each classifier. New models have emerged that tackle this problem from an optimization point of view, such as testing multiple combinations and select the best performing, to more complex methods that use probabilistic algorithms

(38)

22 2.6. MODEL SELECTION

to explore the possible combination space in an eﬃcient manner.

The concept of model selection encompasses a series of independent but related problems; a) selecting the best features, b) selecting the best preprocessing, c) selecting the best classiﬁer, d) selecting the best set of hyper-parameters. The methods that perform optimized selection of all mentioned problems is known asFull Model Selection (FMS).

Feature selection is the process of selecting a subset of features from a feature set, without signiﬁcantly reducing the performance of a recognition system [Pudil et al., 1994]. Filter models evaluate features without utilizing any classiﬁcation algorithms [Liu and Motoda, 2007], this process ranks features according to a criteria, such accuracy=((T P+

T N)/(T P +T N +F P +F N)) 1_{, or mutual information [}_{Koller and Sahami, 1996}_;_Yu

and Liu, 2003; Peng et al., 2005]. As these features are selected without any classifier in mind they undermine the effect of the classifier on the selected features in contrast. Wrapper models [Kohavi and John, 1997; Inza et al., 2004] overcome the previously mentioned shortcoming by pre-selecting the classifier algorithm, thereby being able to evaluate a given feature set and the impact of the classifier, iteratively selecting and evaluating features to obtain a subset, appropriate for the selected algorithm. Imbedded models [Liu and Yu, 2005; Saeys et al., 2007; Ma and Huang, 2008] embed feature selection with classifier construction. These have the advantages of wrapper models as they interact with the classification model and those of filter models as they are less computationally intensive than wrapper methods. The most common embedded method uses pruning algorithms that initialize with all features and then eliminate features as long as performance is maintained. Apart from feature selection the selection, of an appropriated classifier for a feature set is also explored starting with the use of ensembles of classifiers [Kittler et al., 1998;Caruana et al., 2004]. In this method multiple classifiers (as diverse as possible) are trained with the same dataset and selecting those that perform best for that specific problem. These selected classifiers are later combined under the assumption that the overall performance surpasses individual classifiers by

(39)

reducing the effect of individual "independent" errors [Sharkey, 2012]. The problematic with this method is twofold. First is that the assumption of "independent" errors only affect individual classifiers, which is not always the case according to [Giacinto and Roli, 1999]. Secondly, is the problematic of computational requirements of training a large number of different classifiers. As an alternative method [Ho et al., 1994] present the concept ofDynamic Classifier Selection (DCS) were the classes are ranked instead of being unique class choices, these ranking is then used to predict the best classifier for a given input range, to quote [Ho et al., 1994] "a set of mutually exclusive conditions that divides a training set into several partitions. Classifier performance is measured separately on each partition so that the best classifier for each partition is determined".

In general FMS tries to find an optimized solution exploring combinations of all the mentioned elements. This can become computationally inefficient rather quickly making it more cost efficient to use probabilistic approaches.

The method we used in this work, Particle Swarm Model Selection (PSMS)[Escalante et al., 2009] consists of an algorithm that searches through the N dimendional solution

space using a swarm of particles that explore this space and testing solutions at their current "location" and traveling in the directions of their best solution so far while being attracted to the best global solution found. This combination of an own local maximum and a global maximum has a high probability of ﬁnding a good combination of features, classiﬁer, hyper-parameters at a much lower computational cost than the exploration of the entire solution space.

2.7. Chapter summary

Stroke is a medical condition were blood flow to a group of neurons is disrupted, killing those neurons that spend to mush time without oxygen. If this damage affects a region that controls motor skills the victim loses his ability to perform certain movements. To reduce the long term effects medics recommend therapy, a process were movement skills are adjusted to recover certain autonomy. Therapy requires systematic repetition of

(40)

24 2.7. CHAPTER SUMMARY

movements to adjust for the loss of neurons. Videogames and robotic technologies have been used to support this process by presenting alternatives to traditional therapy.

The use of these new technologies has also increased the possibilities of sensing patients during therapy. This allows the creation of representations that map information from sensors to a more manageable format using different projection techniques. Several projection options exist, were the data is adjusted using transformations to obtain a better separation of different classes. Among these projections t-SNE presents a great advantage given the separability it favors allowing classifiers to "easily" map new samples to an existing class.

When a given sample can belong to more than one class at a time, it is necessary to use more complicated approaches. One of such approaches is multi-label classification were different classifiers exist for each of the possible classes and information is shared between them to increase classification rates. But as the number of classifiers increases it becomes harder to select the appropriated classifier for a given dataset. To cope with this optimization, algorithms have emerged that using different metrics select the optimal classifier, or approach the optimal classifier by using faster probabilistic methods.

(41)

Chapter 3

Related work

A review of literature was performed about existing methods to perform automatic motor dexterity assessment using a clinically validated instruments.

3.1. Physical activity monitoring

Physical Activity (PA) is defined by [Caspersen et al., 1985] as any body movement performed by skeletal muscles, that in return results in an energy consumption. The monitoring of PA has been essential in the research of the relationship of health and human movements, specially in the area of cardiovascular diseases, diabetes mellitus and obesity. This is based on the evidence of [Steele et al., 2000] that PAreduction is a major risk factor in multiple illnesses. Multiple methods of assessingPA have been developed including the subjective type such as questionnaires, diaries and surveys, which are low cost solutions but have a low reliability as they depend on observations and subjective interpretation resulting in inconsistent assessment according to [Meijer et al., 1991]. To bring some standardization to PAdifferent tools have been proposed including "timed up-and-go test" by [Podsiadlo and Richardson, 1991] and the "Berg Balance Scale" by [Berg et al., 1989]. These methods even being standardized still have a subjective factor affecting the outcome of the assessment.

To overcome the subjectivity of assessment the use of sensors for PA assessment was introduced, the most common wearable sensors are: pedometers, goniometers, accelerometers and gyroscopes, while sensing environments relay on magnetic, proximity, optical systems. Among the most common sensors used, pedometers are the cheapest

(42)

26 3.1. PHYSICAL ACTIVITY MONITORING

alternative but they lack precision for most applications [Saris and Binkhorst, 1977]. The use of accelerometers have provided plenty of evidence of their reliability for precise long term monitoring. Accelerometry is a tool that is suitable for long-term monitoring of free-living subjects because it can provide objective, reliable monitoring of unconstrained subjects for low cost from accelerometry data alone [Mathie et al., 2004]. A wide range of measures, including classiﬁcation of movements, assessment of physical activity level, estimation of metabolic energy expenditure, assessment of balance, gait and sit-to-stand transfers can be reliably obtained. The use of other non-obtrusive technologies has also been researched using optical sensors such as cameras. These methods heve proven to be good clinical instruments as they give a deeper insight of the PA performed, but they are costly and require special environmental conﬁgurations, combined with privacy issues which limit they use for daily monitoring.

3.1.1. Automatic motor dexterity assessment

The assessment of motor dexterity is a subcategory of PA assessment, with the specific purpose of determining the quality of movement [Morse and Field, 1995]. In post stroke evaluation this quality of movement determines how much autonomy a patient has, to perform daily activities or how similar his movements are in comparison with an average healthy person. Different approaches by [Quintana et al., 2008;Edgar et al., 2010;Bento et al., 2011] have been tested in laboratory and clinical settings, getting promising results in laboratory but generally being insufficient for clinical usage (e.g. not meeting the inter-rater reliability of experts, being to obtrusive according to [Gemperle et al., 1998] to be used by patients, or using not clinically validated scales for their scoring system). As with other physical activity monitoring systems, different approaches in sensing geometry have been used, mainly IMU’s, video tracking, and RGB-D sensors. The systems presented by [Balasubramanian et al., 2008] that relayed on robotic tracking and IMU’s to adjust an exoskeleton to the patients movement. While not directly assessing, the system is capable of adjusting to different motor dexterity levels therefore

(43)

CHAPTER 3. RELATED WORK 27

internally the system does an evaluation of the patient. Other sistems using IMU’s like [Hester et al., 2006; Wade et al., 2010] have generally outperformed visual systems [Allin and Ramanan, 2007;Quintana et al., 2008] by obtaining more reliable data about the orientation of the limb segment being tracked. But are more obtrusive to patients, and harder to be used without clinical intervention. Some studies that usedIMU’s, such as [Stein, 2002; Giorgino et al., 2009] also concluded, during the main or pilot studies, that the correct placement of the sensors played an important role in the performance of their systems, aﬀecting their eﬀectiveness for home based assessment given that the motor disability of the patient might prevent them from correctly positioning the sensors. The use of IMU’s use a combination of information recollected from the sensor including: rotational information from a gyroscope, acceleration and velocity from an accelerometer, and in some cases orientation information from a magnetometer to perform a pre-processing to reduce variance [Parnandi et al., 2010].

The development and cost reduction of RGB-D sensors permitted the development of new methods used to perform automatic assessment since it overcomes the obtrusive factor of other sensors while providing a highly accurate measuring method. This has been confirmed by the works of [Loconsole et al., 2012;Chang et al., 2012] who evaluated the accuracy of the Microsoft Kinect device as a clinical instrument finding it as reliable as high end tracking systems at a fraction of the cost for this purpose. Works that took advantage of this new technology include [Kim et al., 2016] were a maximal accuracy of 87% was achieved for classifying upper extremity exercises scored using FMA. This work relied on the use of PCA for dimensionality reduction and neural networks for the classification.

Of the reviewed works the one that stands out for its similarities with this work, in methodology and results is the one presented in [Otten et al., 2015]. In this work, the author selected a combination of sensors including Microsoft Kinect and glsIMU’s to capture synthetic data from 8 healthy participants acting out diﬀerent motor dexterity levels while performing the upper extremity exercises of FMA. Additionally data from 2 stroke patients was captured using the same setup for clinical validation. The method

(44)

28 3.2. CHAPTER SUMMARY

used consisted of training aSupport Vector Machine (SVM) [Hearst et al., 1998] and a

Backpropagation Neural Network (BNN)[Werbos, 1974] with a set of manually selected features of each sensor to classify the individual exercises. reaching 93.1%±4.0 for

healthy participants and 86.1% for real stroke patients.

Author Year Sensor in-dependent representation

Method Body parts Patients Healthy Clinical scale

[Hester et al., 2006] 2006 No Lineal regresion Upper extremity 12 0 FMA / CMM-SAM /WMFT [Allin and Ramanan, 2007] 2007 No Lineal regresion Upper extremity 7 0 AMAT [Quintana et al., 2008] 2008 No HMM

compari-son

Upper extremity 10 2 MI / FMA

[Edgar et al., 2010] 2010 No Undeﬁned Lower extremities 0 2 NONE [Wade et al., 2010] 2010 No Undeﬁned Upper extremity 0 2 WMFT [Bento et al., 2011] 2011 No Random Forest

classiﬁer

Upper extremity 5 0 WMFT

[Otten et al., 2015] 2015 No SVM Upper extremity 2 8 FMA

Proposal 2016 Yes FMS, multi-label

Upper extremity 23 15 FMA

Table 3.1.: Methods found in literature for the automatic assessment of motor dexterity and the method proposed in this work.

Of the reviewed works none uses a representation that is independent of the sensor. This dependence limits the areas of implementation to clinics that have access to the same exact sensor setup they tested. Also the dependence of the sensing geometry makes those proposed systems unscalable.

It is worth to noting that even given the clinical evidence of relationships among diﬀerent exercises, this has not been thus far exploited by the automatic motor dexterity assessment methods reviewed. This limits for instance the possibility of assessing a patient with a incomplete movement dataset.

3.2. Chapter summary

Existing work on automatization of motor dexterity assessment has focused on methods that can evaluate motor dexterity of a person using diﬀerent artiﬁcial intelligence

(45)

CHAPTER 3. RELATED WORK 29

algorithms that analyze the data from a specific sensor configuration. The major improvements among the methods have come with the use of newer more reliable sensors and the combination of different sources.

From this bibliographical review several things can be concluded;

The importance of an automatic assessment system that reduces specialist burden, and allows an objective way of determining patient motor dexterity and progress. Clinicians do not fully agree on the eﬀect of sensor obtrusiveness, but in general the tendency shows that obtrusive sensors should be avoided when possible. None of the reviewed works has made it clear that their assessment could be performed independently from the sensing geometry they depart from, or even using diﬀerent types of sensors of the same kind. This lack of generalizability makes the assessment process a weak and unreliable solutions for non controlled environments. Finally, there seems to be a tendency for systems that use only synthetic data to outperform systems that combined datasets or only use patient data.

(46)

(47)

Chapter 4

Methodology

As [Bengio et al., 2013] stated "A good representation is also one that is useful as input to a supervised predictor." Under this premise we propose a representation that abstracts the observed phenomenon (FMA exercises) from the sensing technology used to monitor (Kinect, IMU’s), so we can effectively classify the motor dexterity of a patient without depending on a specific sensing technology. For this to be feasible, the representation resulting from a same observation sensed using different sensors should be undifferentiable, while conveying sufficient information about the construct for posterior identification/classification.

Once a sensor independent representation has been achieved the problem of deter-mining motor dexterity becomes a classiﬁcation problem were each exercise is analyzed to determine the class it belongs to. In this case, the classes are those of the FMA

3-point ordinal scoring scale, thus being a multi-class classification problem, with a high number of possible combinations (classifier, parameter, features). Given all the possible combinations, the selection of a model that is optimized for this problem becomes a difficult task, unless a system is used to find the mathematically optimized solution.

The problem of motor dexterity assessment creates another challenge given that it requires a series of independent exercises to be classiﬁed. And even tough these exercises are performed independently one from another, there is bibliographical evidence that the performance of a patient in one exercise can indicate his probable performance at another exercise with witch it shares muscular or neurological structures. This shared

(48)

32 4.1. SENSOR SELECTION

biological structure presents an ideal candidate to improve classification using multi-label classification sharing information among the different classifiers. The final problem for a highly accurate classification then relies on, finding how to share information among classifiers that represents the relationship among exercises.

The general methodology of this work is shown in Fig 4.1. In depth description of each of the steps of the methodology can be found in sections: Data acquisition 4.1, Representation 4.4, Model selection 4.5, Multi-label classiﬁcation 4.6, and Ancestor augmented network-based chain multi-label classiﬁers4.7.

Figure 4.1.: Schematic view of the methodology used for this work.

4.1. Sensor selection

Given the restrictions and the sensors available at the robotics lab of INAOE the decision was made to use three diﬀerent sensors in addition to video recording for posterior clinical review. The selected sensors were:

(49)

CHAPTER 4. METHODOLOGY 33

A low cost range sensor based on range camera technology by Primesense produced by Microsoft for XBox 360 console, developed as a hands-free controller for video games. The sensor consists of a color 8-bit VGA resolution (640 X 480 pixels) camera, a monochromatic 11-bit Video Graphics Array (VGA) resolution (640 X 480 pixels) IR camera, and a calibratedInfra Red (IR) laser grid projector.

The device works by emitting an IR calibrated grid and where the returning rays are captured by the IRcamera and the distortions of the grid on the surfaces of the environment are used to calculate a depth map representing the distances from the sensor to the objects in the field of view. The produced depth map is then used by different software algorithms to detect human shapes and infer the position of different predefined segments of a human body, these segments are then exposed to a system as a virtual 3D skeleton.

At the moment of deﬁning the setup, the Microsoft Software Development Kit (SDK) was not considered reliable for seating person detection, which is a

require-ment for upper extremity FMA, so the desicion was made to use OpenNI software in combination with Primesense NITE SDK library to detect and track subjects while seated. This is shown in Fig 4.2.

Figure 4.2.: Screenshot of OpenNI/NITE capture software, demonstrating the capability of capturing seated persons skeletal structure.

(50)

34 4.1. SENSOR SELECTION

• LP-Research LPMS-B:

The LPMS-B is a miniature wirelessIMU/Attitude and Heading Reference System (AHRS) device for measuring the orientation of the device. The sensor uses three internally located units: a 3-axis gyroscope, which detects angular velocity; a 3-axis accelerometer, which detects linear acceleration in x, y and z; and a 3-axis magnetometer, for measuring direction of the earth magnetic ﬁeld. The IMU communicates with an external computer transmitting data through Bluetooth communication. LPMS- ﬁrmware makes a pre-processing of the sensor signals and provides the host with a text formatted string of values that can be accessed via software.

• Gesture Therapy Gripper Gesture Therapy Gripper, is a custom controller device created and patented at Instituto Nacional de Astrofísica, Óptica y Elec-trónica (INAOE). This device is used as an input device that combines positional information obtained trough video tracking, and information from an integrated pressure sensor. The gripper used in this work is a prototype version still under development. The prototype includes new sensors such as accelerometers, for better interaction and control.

The above sensors are shown in Fig 4.3.

(a) Microsoft Kinect.

(b) P-Research LPMS-B.

(c) Gesture Therapy Gripper.

(51)

(52)

36 4.2. SENSOR CONFIGURATION

For data capture, a bespoken synchronized multi-thread system ilustrated in Fig 4.5

was developed using C++, using OpenNI (1.5.2.23), NITE (x64-v1.5.2.21) for Kinect device control and skeleton tracking, libLPMS (1.2.8) for communication with LPMS-B device, and libBluez (4.101-r9 5.35) for communication with GT bluetooth Gripper. The software captures all sensors simultaneously at 30Hz while also recording video of the color and IR cameras of the Kinect. A total of 40 channels were registered in addition to time stamps and scoring labels, see Table: 4.1 for description. Information from Kinect was also stored in ONI ﬁle format for future skeleton detection alternatives.

Figure 4.5.: Custom data capture software interface. Capture was performed using synchronous muti-thread capture data from all sensors was captured along with video from the patient for future reference. (Tabs in the GUI give access to information relevant for the respective thread).

(53)

CHAPTER 4. METHODOLOGY 37

Sensor Variables Description

Kinect RShoulder-X,Y,Z (Right Shoulder X,Y,Z using torso as reference) Kinect RElbow-X,Y,Z (Right Elbow X,Y,Z using torso as reference) Kinect RHand-X,Y,Z (Right Hand X,Y,Z using torso as reference) Kinect LShoulder-X,Y,Z (Left Shoulder X,Y,Z using torso as reference) Kinect LElbow-X,Y,Z (Left Elbow X,Y,Z using torso as reference) Kinect LHand-X,Y,Z (Left Hand X,Y,Z using torso as reference) Kinect Torso-X,Y,Z (Torso X,Y,Z using Kinect center as reference) Kinect Head-X,Y,Z (Head X,Y,Z using torso as reference) Gripper BTGripper-X,Y,Z (Gripper X,Y,Z absolute acceleration) Gripper BTGripper-RX,RY,RZ (Gripper X,Y,Z absolute rotation) LPMS-B LPMS-QUAT-W,X,Y,Z (LPMS absolute rotation as quaternion) LPMS-B LPMS-ACCEL-RAW-X,Y,Z (LPMS X,Y,Z absolute acceleration) LPMS-B LPMS-GYRO-RAW-X,Y,Z (LPMS X,Y,Z absolute rotation)

Table 4.1.: Breakdown of variables captured by custom capture software.

All data capture was performed on a Acer Aspire V5 laptop with a Intel i5 processor, 8GB Ram, and 500GB hard drive running gentoo. A more detailed description is given in Table4.2

Operating system Linux 64 4.0.5-gentoo Desktop manager KDE 4.14.16

Hard Drive 500GB Western Digital

Ram 2/4GB DDR Kingston

Processor Intel(R) Core(TM) i5-3317U CPU @ 1.70GHz GPU Intel Gallium 0.4 LLVM 3.5

Table 4.2.: Description of system used for data capture.

4.3. Signal segmentation

Given that the data was continuously captured during an entire FMA session, the signal from the sensors had to be separated in segments encompassing single exercises. Therefore the raw signal was segmented by means of a semi-supervised segmentation algorithm developed in Matlab that presents a GUI shown in Fig4.6. This GUI displays

(54)

38 4.4. SENSOR ABSTRACTED REPRESENTATION

one sensor channel at a time allowing the user to select the channel with the keyboard. Then the algorithm automatically searches the probable start/stop points for the exercise displaying the cut-points in red while showing the sub-segment in a secondary window were the user can ﬁne tune the selection via keyboard. Finally the user can save the segment to a new ﬁle, labeling it during the process. Finally, all data was linearly scaled to normalize the readings and saved in cvs format.

Figure 4.6.: Bespoken segmentation software developed in Matlab for the segmentation of raw data signals in a semi-supervised fashion.

4.4. Sensor abstracted representation

To partially address our hypothesis, we propose a representationR that is nonspeciﬁc

across a family of motion amenable sensors 1 _{to maintain the classiﬁcation problem}

1_{Full abstraction from the sensing geometry is beyond the scope of this work. For instance, we do not}

(55)

(56)

Automatic assessment of motor dexterity for rehabilitation with a representation nonspecific on sensor