Registro Contable

HAR capabilities within a smart environment pose many challenges at all the five phases defined in section 2.1.2. Firstly, the data collection phase consists of employing diverse sensing methods which has several challenges and open issues that need to be addressed such as privacy, security, practical, interoperability, technical challenges and financial implications. Section 2.7.1 elaborates on some of the challenges to monitor and capture the inhabitants' behaviours using available sensing techniques. Secondly, the data segmentation phase requires sensor events to be disentangled into a relevant set of ongoing ADLs despite the order of actions and mixed activities; more details are provided in section 2.7.2.

Thirdly, data modelling phase has several challenges to address, such as intricately describing ADLs, complex smart sensing infrastructure, environmental entities, and user profiles for data segmentation and activity recognition (fourth) phase. Further challenges are created when modelling imprecise non-binary sensor data and several uncertainties caused when performing mixed activity recognition accurately. Section 2.7.3 expands on problems related to mixed activity recognition while sections 2.7.4 and 2.7.5 discuss challenges in modelling fine- grained user actions with imprecise sensors and uncertainty factors influencing the activity recognition accuracy.

Fourthly, the activity recognition phase is influenced by the KD or DD activity modelling approaches selected to recognise single and multi-user activities. Section 2.7.6 elaborate on challenges of detecting, identifying, and associating user actions in the shared living environment. Lastly, the activity learning phase needs large datasets that are well- formatted and annotated to perform pattern, and frequency algorithms developed using supervised, or unsupervised DD approaches. Nevertheless, investigating on challenges related to activity learning phase is out of the scope for the thesis as discussed in section 1.4.

For each of the abovementioned AR phases, various interdependent underlying technologies also exist [36]. These technologies present further system architectural integration challenges, which are mainly due to their differences in programming languages, development environments, proprietary components, data storage and communication protocols. Therefore, the interconnectivity of each phase into a single platform poses several challenges when developing a unified system architecture styles and patterns tailored to the AAL system. These AAL system architectural level and big data storage challenges are elaborated in sections 2.7.7 and 2.7.8.

2.7.1. Multimodal Smart Sensing Environment

The vision-based approach has been used extensively for security and surveillance systems. However, concerns with the privacy of inhabitants in their private homes have led researchers to explore unobtrusive and pervasive sensor-based approaches.

Another challenge is that several vendors make application-specific off-the-shelf products that are not always open-source and run on diverse communication protocols. Hence, creating a big problem when integrating these cross-manufacturer devices within WSNs of any given size. However, to address this challenge, many efforts have been exerted by the vendors in recent years. One common practice is to provide application program interfaces (APIs) and software development kits (SDKs) to allow cross-platform third-party service integrations. For instance, Securifi Almond+ router, Amazon Echo [122], and Samsung SmartThings [123] can interact with each other's devices. Although these services are growing, limited intelligence can be added to the sensor nodes as rules govern them, such as “if this, then that” concepts (i.e., IFTTT [124]). Furthermore, they still have limited types of sensors that can support fine-grained sensing capabilities for AR, i.e., a capacitive touch sensor on an object for dense sensing. Therefore, bespoke Arduino-based wireless sensing methods are still commonly used [125], [126]. Therefore, this thesis investigates on the challenges to integrate some of those above off- the-shelf and open-source WSN technologies within the AAL system architecture to achieve real-time AR, monitoring, and assistance provisioning.

2.7.2. Sensor Data Segmentation

One of the critical challenges for segmenting data continuous sensor data stream is to disentangled single and mixed activities with actions conducted in any order. Also, most studies model generic set of actions for ADLs when segmenting continuous data and performing AR. However, users in real-world are likely to have personal preferences on conducting ADLs with unique ingredients or a variety of utensils. Hence, creating a challenge to not only model generic and user-preferences but also incorporating the model in the data segmentation process. 2.7.3. Mixed Activity Recognition (AR)

Activity recognition (AR) phase rely on the segmented sensor data and ADL knowledge model to recognise single and mixed user activities. Developing the ADL model with KD approach need domain expert knowledge and manual effort to maintain/evolve the model compared to the DD approach requiring large pre-collected dataset to train the model. KD model is reusable with other users and can be shared across domain is contrary to the model developed with DD. Both approaches can only define a finite set of activities in the model. Hence, the hybrid approach must use the initial model developed in KD and evolve with the DD approach.

2.7.4. Fine-grained Action Detection with Multimodal Sensor Data

Recent studies recognise activities by assuming completion of an action using binary sensors. However, more attention is required to fuse multiple sensing attributes of an object to recognise user interactions as fine-grained level. However, non-binary sensors data from the smart environment create impreciseness and subjective interpretation for the status of the object, i.e., if the cup is “low” or “half full”. Hence, the fine-grained action modelling approach is required for ADLs to not only define imprecise concepts but also fuse sensing attributes.

2.7.5. Uncertainty Factors in AR

Several problems related to technology failure, human error, environmental condition and object functionality can affect the reliability and trustworthiness of the observed sensor data and AR results. For instance, a low battery level of a sensor can impact the sensor reading accuracy and transmitting range due to a weak signal, which can ultimately cause error, delay or loss of packets. Thus, influencing the AR results. Hence, uncertainty theories such as probabilistic [127], [128], evidential [129], [130], and fuzzy [57], [131] need to be investigated to model and reason with the uncertainty factors present in AR.

2.7.6. Multi-user AR in Shared Living Environment

AAL systems are likely to be deployed in a dwelling with more than one occupant. Hence, some of the main challenges are to detect if there are multiple occupants in the same environment, identify the occupant using discriminating sensing techniques and associate their actions with ongoing ADLs[28]. Therefore, a multi-user AR approach is needed with a smart environment equipped with non-obstructive sensing approaches that preserve the privacy of the occupant and security of the personal data.

2.7.7. AAL System Architecture Style and Patterns

One of the main challenges in building an assistive system is to select appropriate system architectural styles and patterns which can be easily misused [132]–[134]. Engaging with the broader community by having open source components and using popular programming languages can play a crucial role in coming up with useful, adaptive, and personalised solutions. Other factors influencing the design decisions include semantical data storage, computation power requirement, low latency communication protocols, and the ability to allow simultaneous access to the users with a convenient human-computer interface (HCI). Some of the exiting assistive systems (explored further in section 8.2) are built in a standalone application environment. However, questions have been raised regarding its extensibility, reusability, scalability, maintainability, and/or use of proprietary components, which may have limited

community support. In addition, having a poor or an unnatural HCI design poses practical limitations for its key users.

Over the years, the service-orientated architecture (SOA) approach has become popular, because it can address some of the aforementioned issues as well as create a mechanism by which to delegate resource-intensive tasks and storage to powerful sets of computers over a network (cloud computing). Moreover, using the SOA approach also allows low-power devices such as mobile devices or any other gadgets with network capabilities, to utilise the available services. This has not only improved the HCI of the AAL system but also made it scalable such that it can serve cross-platform clients as well as integrate and reuse third-party services in a creative manner. SOA approach now drives the concepts of SH, IoT, and ubiquitous or pervasive computing. This is the main approach by which everyday objects can be seamlessly integrated into the interconnected World Wide Web (WWW).

2.7.8. Semantic Data Storage

Another challenge faced that arises from this topic are the problem of storing the activity modelling and recognition data using the semantical structure that can be used in a meaningful way. The storage options considered here also influence the overall system architectural design decisions. Recently, this has become a much broader issue with the accumulation of large amounts of unstructured or in semi-structured data with no clear semantical relations. Hence, creating many problems such as automating the task of processing and retrieving data efficiently [70]. Currently, machine-learning techniques, such as genetic algorithms are used to extract and train computers on how to process the data over time. This approach, however, is lengthy and needs a high computation rate.

For efficiency, the concept of the semantic web was introduced. This concept was initially envisioned by Tim Berners-Lee and his colleagues for creating the WWW with linked data structure with semantic meanings defined using formal methods that can be processed by a machine [71], [135]. Hence, representing the data in the form of a triplet, subject-predict-object to form a connected graph. The most common vocabularies are used and shared to create an expressivity in the data (i.e., using Resource Description Framework (RDF) [136], [137] and Web Ontology Language (OWL) [138]). Also, various reasoning engines (i.e., Pellet, HermiT, and FaCT++) are used to perform inferencing utilising the user-specific rules and formal languages. The set of triplets are stored in the triplestore (database) as a graph, which is specially optimised for handling them. Moreover, just like the Structured Query Language (SQL) in traditional relational databases, the SPARQL Protocol and RDF Query language (SPARQL) are used to perform, create, read, update and delete (CRUD) operations [137], [139].

These capabilities and benefits enable the back end of any application to achieve greater flexibility within its specific system architecture.

2.8. Summary

This chapter presents a research background of HAR with five key phases, reviews of state-of- the-art studies related to these phases, and highlights of critical challenges, opportunities and open issues identified as a result. The complementary relationship between the five HAR phases to develop a suitable system architecture based AAL system requirements and integration of multimodal smart environments are discussed.

The following chapters will investigate on eight critical challenges identified in section 2.7, propose and evaluate novel methods, approach and framework. CHAPTER 3 examines on data segmentation challenges described in sections 2.7.1 and 2.7.2 to utilise semantical relationships of the sensors to associate a set of actions to a given ADL. CHAPTER 4 builds on the challenges of recognising mixed activities at multi-granular levels by fusing multimodal sensors data and ADLs model as described in sections 2.7.1, 2.7.3 and 2.7.4. CHAPTER 5 investigates on the uncertainty factors that exist in recognising ADLs accurately as described in section 2.7.5. CHAPTER 6 presents an overall framework for single AR, and CHAPTER 7 extends to identify and associate multi-user actions within a shared smart environment, as highlighted in section 2.7.6. CHAPTER 8 analyses the state-of-art system architecture for AAL systems and proposed a suitable microservices architecture with graph-based big data storage requirements discussed in sections 2.7.7 and 2.7.8. Finally, CHAPTER 9 summaries each chapter, presents key contributions made in this thesis, discuss challenges and open issues to be addressed in future work and offer concluding remarks.

CHAPTER 3.

SEMANTIC-ENABLED

SENSOR

DATA