FASE I. Planificación Preliminar

Several data-driven (DD) and knowledge-based (KB) [20], [34] approaches have attempted to develop fine-grained action level AR by integrating imprecise/fuzzy sensor values[168], [169] and/or uncertainties[169]. These approaches frequently adapt mathematical theories such as fuzzy [166], [168], [172], probabilistic [37], [173], [174], possibilistic [175], [176], and Dempster-Shafer (DF)/Evidential theory[169] to model and reason with multimodal sensor data. In the following sections, studies are reviewed related to DD and KB approaches for AR at a fine-grained level in sections 4.2.1 and 4.2.2.

4.2.1. Fine-grained Level AR with DD Approaches

To achieve fine-grained AR with DD methods, work in [177] combined acceleration, acoustic and multi-sensor classifiers. These classifiers are J48 decision tree, random forest (RF), a Bayesian network, and support vector machine (SVM). A single off-the-shelf smartwatch was used in the experiment to recognise five daily activities, i.e., eating, vacuuming, sleeping, showering and watching TV. The result indicates that the combined approach achieved greater accuracy (91.5%) in comparison to individual classifiers. The main shortcoming of this approach is that the measurement of the sensors is not compared against the degree of use (imprecise). For instance, what is the “minimum” wrist movement required to infer the vacuuming action and if the watch is left on the table facing “up” before going to the shower compared to falling asleep with folded hands.

Work in [170], adopts weight-based probabilistic and conditional random field (CRF) decision classifiers with multimodal and multi-positional (wrist, back, leg and waist) sensors to achieve 80% AR accuracy of 19 coarse-grained and fine-grained routines in daily living. Likewise, work in [178] used an inertial ring and a bracelet to achieve fine-grained occupant activity recognition based on the wrist and index finger gestures of eating, drinking and brushing with favourable initial results. The limitations for both approaches are the ability to automatically link wearable sensors on the body part with gestures and embedded sensors with everyday objects. Furthermore, the inherently obtrusive nature and limited battery lifespan of wearable sensors create challenges for the widespread adoption of the system. Consequently, work in [179] developed a passive RFID based Moo Tag with onboard 3-axes accelerator sensor to attached to non-/perishable objects with ultra-high frequency RFID reader to detect fine- grained user action. The tag ID, Received Signal Strength Indicator (RSSI) and accelerometer values from the passive sensor tags are in congestion with HMM model to infer fine-grained

actions. Nevertheless, these passive Moo Tag have limited computational, data storage and transportation capacity to attach more sensor data to increase the accuracy to determine completion of an action.

Work in [175], explored knowledge-based possibilistic network classifiers to handle uncertainty (imprecise, incomplete, missing) in sensor data when taking medication (with get water and take the pill as fine-grained actions) in AAL setting. Though, this approach still assumes that interaction with the everyday object as part of key sub-/action is the satisfactory complication of action. For instance, getting a cup and turning the tap on does not always mean the cup is being filled or “minimum” quantity of water is filled in the cup correctly. Therefore, additional sensors such as liquid level, accelerometer and gyroscope are required in the cup to be correlated and validate “getting water action”. Additionally, limited support is shown to handle imprecise raw sensor data such as water level in the cup, and if the user has drank the water when detecting fine-grained action.

4.2.2. Imprecise Measurements with Knowledge-based AR

The knowledge-based approach initiates the modelling process by the formally conceptualising intricate knowledge by a domain expert(s). This knowledge model overcomes the “cold start” issue and increases reusability by modelling activities at multiple levels of abstraction. Nonetheless, the models created with knowledge engineering techniques require manual efforts[180], limited to the domain expert’s knowledge, and incomplete.

In the KB approach, Web Ontology Language (OWL) is a backbone of semantic web language. OWL enables the formal representation of rich and complex knowledge by the domain expert(s) that can be reusable, human-readable and machine friendly. The ontology modelling techniques have been extensively leveraged to conceptualise concepts, describe relationships using a family of description logics (DLs) and reason with the explicitly defined information to deduce inexplicit knowledge. Yet, OWL and DL suffer from the ability to support imprecise/vague concepts.

An example of the study can be seen in [181], which presented a multi-level context- aware recognition framework(mlCAF). This framework developed a cross-domain (physical activity, nutrition, and clinical) ontological model and Web Rule Language (SWRL) rules-based reasoning. The low-level (fine-grained actions) contextual information such as nutritional and behavioural patterns of the inhabitants is initially inferred using cross-domain ontology-based inferencing with the support of the Pellet reasoner. The high-level context (coarse-grained activity) based on human behaviour and lifestyle is determined by using SWRL/SQRWL rules which keeps on making associations between three domains and low-level context at different

levels. Similarly, work in [182] uses ontology and bespoke SPARQL Protocol and RDF query language (SPARQL) to recognise activities at two granularly levels. Nevertheless, both approaches suffer the same issue and add the complexity of manual querying.

The recent studies have extended the OWL expressivity capabilities and incorporated imprecise/vague concepts with the fuzzy ontology. The fuzzy ontology is based on fuzzy set theory. The fuzzy set theory allows one to associate a fuzzy concept with having a degree of membership in a given set by defining one value (Type-1) or two values between 0 and 1 (Type- 2) [172]. Work in [166] developed a standalone Type-1 fuzzy logic system to recognise around 18 coarse-grained ADLs and human body motion. The types of sensors used in the system are physiological sensors, microphone, infrared sensors, debit sensors (for water flow) and state- change sensors. The system was developed using Labwindows CVI and C++ software. The preliminary results show that 97% accuracy in recognising ADLs. The fuzzy theory has been adopted to support decision making and combining multiple sensor data when recognising ADLs using ontology[183]–[185] and in other domains such as flight booking[172], and diabetic mellitus[171]. The common problems of these fuzzy ontology-based studies are the lack of emphasis on accurately detecting fine-grained actions based on object usage. Furthermore, there are limited tools available to develop fuzzy ontology and perform automatic reasoning. Though Umberto and his team have recently developed a fuzzy ontology plugin for Protégé [186] (ontology editor), and fuzzyDL[187] reasoner; see [188] more details. To the best of our knowledge, fuzzyDL plugin and reasoner have not been evaluated for detecting fine- grained AR within a real-time distributed system.

This chapter focuses on making four main contributions to recognising activities at the multi-granularity level. The first contribution is the approach to model coarse-grained and fine- grained actions required for ADLs using KD approach. The model at the coarse-grained action level consists of capturing complex context, environment, and relationships between everyday objects. Likewise, at the fine-grained actions level, everyday objects and their changing states are modelled with multimodal sensor (i.e., liquid level, temperature, accelerometer, and gyroscope) readings. The second contribution is the approach to represent the imprecise nature of some non-binary sensor readings into fuzzy concepts/state of a given object (i.e., kettle water temperature is “hot”). The third contribution is the approach to fusion multimodal sensor readings to detect fine-grained actions. For instance, pouring action for a kettle can be defined when the temperature is “hot”, the liquid level is “full” and gyroscope Z value is “tilt”). The fourth contribution is the decision engine that progressively takes multimodal sensor readings and multi-granularity knowledge model as inputs to calculate the degree of action completion.

The decision engine algorithm has been developed and evaluated in a distributed prototype system.