The detection of a fall is positive if the fall detection system can distinguish a fall and negative if it cannot. As the output is in binary format, the quality of the detector cannot be evaluated through one single simple test. Therefore, it is crucial to analyse the results using some statistical tests. There are four possibilities in the detection of a fall: a true positive (TP) is returned when a fall occurs and the sensor detects it; a false positive (FP) is when the fall sensor identifies a fall where in fact a fall did not occur; a true negative (TN) is where only normal movement occurs (there is no fall) and the sensor does not detect a fall; and a false negative (FN) is returned when a fall occurs but the fall detection system fails to detect it. In order to evaluate the responses in terms of these four possibilities, two criteria are assessed: sensitivity and specificity, which are normally used for evaluation purposes. Sensitivity is the capacity of the fall detection system to detect and declare a fall, while specificity is the capacity to detect only a fall. Equations 4.6.1 and 4.6.2 were used to calculate specificity and sensitivity respectively, while Table 9.2 shows the definitions of TP, FP, FN and TN.
Fall incident/System Recognition Fall occurs Fall not occurs
Positive TP FP
Negative FN TN
Table 10.2: Definitions of TP, FP, FN and TN
There are several possible falling scenarios, but a limited number of positive and negative fall situations were used for the evaluation of the fall detection system. Most of the falls that occurred in this research were either in the anteroposterior
CHAPTER 10. EVALUATION OF THE FALL DETECTION SYSTEM
plane, consisting of both forward and backward slips, or sideways falls. There are many motions in daily life in which the intensity of the movement can be the same as in accidental situations, such as the actions of lying or sitting down quickly, which can imitate falls. Therefore, Noury [128] and Bourke [55] propose a set of scenarios to determine the sensitivity and specificity of a fall algorithm, shown in Table 9.3.
Category Name Outcome
Backward fall Ending lying Positive Ending in lateral position Positive With recovery Negative Forward fall Ending lying flat Positive
With rotation, ending in the lateral right position Positive With rotation, ending in the lateral left position Positive With recovery Negative Lateral fall to the right Ending lying flat Positive
With recovery Negative Lateral fall to the left Ending lying flat Positive
With recovery Negative Neutral Sitting down on a chair Negative Walking a few metres Negative Bending down to collect something and rising up Negative Table 10.3: Scenarios for the evaluation of fall detectors
The first column in Table 9.3 represents the fall categories: backward fall, for- ward fall, lateral fall to the right, lateral fall to the left and neutral (representing a non-fall situation). The second column describes the ending of the fall: whether the person ends lying down flat, on the side, with or without rotation, or recovers from the fall (which means that he or she stands up and continues activity). The
CHAPTER 10. EVALUATION OF THE FALL DETECTION SYSTEM
third column shows the expected outcome of these fall or non-fall situations, which can be either positive, which means a fall occurs, or negative, which means that no fall occurs. These expected outcomes are then used to evaluate the fall detection algorithm, such that corresponding probabilities for sensitivity and specificity are determined from the expected outcome (theoretical outcomes are normally 100%) and the real outcome from the fall detection algorithm.
In order to evaluate the fall detection algorithm for these evaluation scenarios, one realisation of each scenario per participant was not enough. The participants had to perform several scenarios three times in different orders and were able to rest between tests. Each subject also had to vary his speed in performing any pre- defined scenarios as determined by the researcher. To prevent any learning effect or habituation to the gesture following the consecutive reproduction of the same scenario, the order of the tests was varied. Of the 14 scenarios listed in Table 9.3, half are classified as positive (falls) and half as negative (non-falls). As each subject performed three trials for each condition, then in total each subject did 42 tests, making a grand total of 420 tests for the sample of 10 subjects. These 420 data points are considered sufficient to provide a statistically significant computation of the specificity and sensitivity of the device. It is worth noting here that the subjects ranged in age from 20 to 40 years and were all in good health. It was not possible to sample the target population of elderly people, as this would have involved many additional ethical considerations because of the increased health and safety risks, which would have meant that a medical practitioner would have had to attend the experiments.
Table 9.4 shows the percentage of falls and non-falls detected for each scenario; it was found that a strong majority of the scenarios were successfully classified, with
CHAPTER 10. EVALUATION OF THE FALL DETECTION SYSTEM
an average of 94.3%. An ideal fall detection system would be able to produce 100% sensitivity and 100% specificity, which can be obtained during the setting up of ex- periments, but there will be significant loss of performance during the design of an autonomous fall detection system, which may be due to unwanted artefacts in the extracted silhouette.
Category Name Outcome(correct detection/ total)
Percentage of Outcome
Backward fall Ending lying 27/30 90% Ending in lateral posi-
tion
30/30 100% With recovery 30/30 100% Forward fall Ending lying flat 30/30 100%
With rotation,ending in the lateral right po- sition
27/30 90% With rotation,ending
in the lateral left po- sition
29/30 96.7% With recovery 30/30 100% Lateral fall to
the right
Ending lying flat 30/30 100% With recovery 29/30 96.7% Lateral fall to
the left
Ending lying flat 26/30 86.6% With recovery 27/30 90% Neutral Sitting down on a
chair
26/30 86.6% Walking a few metres 30/30 100% Bending down to col-
lect something and ris- ing up
26/30 86.6%
Table 10.4: Percentage of detecting a fall or non-fall on normal gait
In total, it was found that there were 199 TPs, 11 FNs, 197 TNs and 13 FPs (Table 9.5 and Figure 10.1). These figures were used in equations 4.6.1 and 4.6.2 to give a specificity of 94.8% and a sensitivity of 93.8%. Despite the good overall
CHAPTER 10. EVALUATION OF THE FALL DETECTION SYSTEM
performance of the fall detection device, these metrics could be improved, reaching 99% to 100%, by taking into consideration various factors such as the physiology and kinematic parameters of the moving individuals or by analysing their gait (way of walking).
Fall incident/System Recognition Fall occurs Fall not occurs
Positive 199 13
Negative 11 197
Table 10.5: Evaluation of recognition results
CHAPTER 10. EVALUATION OF THE FALL DETECTION SYSTEM