A CTIVITATS D ’E MPRENEDORIA

We are particularly interested in applying this technique to the healthcare training domain as a complement to existing patient simulator technologies, for which integrated touch input and re- sponse is generally not available. This is in spite of the fact that touch is a common component of patient care, including for diagnostic or comfort-related purposes (summarized in Figure 1.5), sug- gesting the value of supporting touch as an input modality for patient simulators. While the ability to touch a simulated patient is itself useful, the primary benefits arise from the manner in which

the simulator responds to or exhibits awareness of touch. In fact, in an exploratory study, we found that even external, manual control of virtual patient touch awareness by an observer monitoring trainee behavior over a video camera—often referred to as a Wizard of Oz paradigm [37]—can be a valuable component of a patient simulator [34]. However, in this dissertation, we are focused on supporting automatic touch detection and response, particularly for the kinds of touch input that are not practically achievable in such human-in-the-loop paradigms due to the level of accuracy, responsiveness, and dynamic interactivity needed—for instance, if a nurse needs to examine a patient’s teeth by manually moving his or her lips or touch a patient at extremely precise locations.

✔ ✔ Rub Push Differentiate ✔ Object ✔ Warm/Cool Palm Two Hands Massage Tap ✔ ✔ Measure ✔ ✔ ✔ ✔ Pinch/ Spread Palpation ✔ ✔ Grasp ✔ Stroke/ Wipe ✔ ✔ ✔ Examine Multiple Fingers ✔ ✔ ✔ ✔ Diagnosis Therapy Purpose Type

Figure 1.5: Common types and purposes of touch in patient care. Table contents developed to- gether with (alphabetical) Kelly Allred, Laura Gonzalez, Mary Lou Sole, Steve Talbert, and Gre- gory Welch.

Rather than targeting this application exclusively, this dissertation presents a generalizable ap- proach that is suitable to a variety of scenarios, including healthcare training. However, this spe- cific motivation informed some of our design decisions, and so we briefly cover relevant aspects. Before we describe typical healthcare simulators, we first introduce a few important definitions.

Lombard and Ditton define presence as “the perceptual illusion of nonmediation” [95]. They further expand this notion as follows:

An “illusion of nonmediation” occurs when a person fails to perceive or acknowledge the existence of a medium in his/her communication environment and responds as he/she would if the medium were not there.

This concept has important implications on healthcare training, which often relies on patient simu- lation: as the presence of a particular medium increases, training can be more effective [41], users are more likely to treat the medium as a social entity, and user behavior becomes more typical of human-human interactions.

Other researchers have extended this definition, examining related and more specific types of presence. Biocca et al. define social presence as “the sense of being with another” [14]. Co- presence has been described as a person’s perception of another person and their sense of being perceived by the other person [56]; Harms and Biocca consider co-presence to be one of six sub- dimensions of social presence [62]. Furthermore, the physicality of an agent or avatar includes the fidelity of its physical occupancy—attributes such as size, shape, and position—and its ability to change or sense changes in the surrounding environment [28]. Increasing the physicality of agents and avatars has been shown to increase social presence and lead to more realistic behavior from users [29, 78, 89, 94, 123].

From these findings, there are immediate benefits arising from the methodology presented in this dissertation. When representing simulated patients, non-parametric surfaces afford increased phys- icality, which results in increased social presence. Likewise, surfaces of any shape and size can exhibit greater awareness of the environment with the addition of touch-triggered behavior, which further increases physicality and thus social presence. Such increases ultimately promote more realistic trainee behavior and thus more effective and engaging training.

Typically, patient simulators in modern healthcare training can be categorized as standardized pa- tients, computer-based simulators, mannequin-based simulators, or mixed-reality simulators [93].

• Standardized patients are humans who have been trained to simulate a specific patient or patients in a consistent, replicable manner [96]. Along with mimicking symptoms, standard- ized patients present an accompanying patient history and simulate the various physical and emotional behaviors that an actual patient representing these symptoms would exhibit. Sim- ulations with humans naturally benefit from an extremely high degree of realism, physical presence, and social presence. However, certain situations preclude the usage of standard- ized patients, such as symptoms that are difficult to realistically replicate or scenarios that involve pediatric patients.

• Computer-based simulators instead portray these symptoms and other aspects through com- puter screens and audio speakers [96]. Compared to standardized patients, computer-based simulators can be advantageous due to their ability to display dynamic imagery, potentially allowing for the portrayal of complicated visual symptoms. One notable disadvantage is the lack of physicality resulting from the two-dimensional representation of the patient, which may make simulations less compelling or prevent learners from treating the simulated pa- tient in a realistic manner. As they are controlled via software, computer-based simulators can provide extremely consistent experiences through specific predefined behaviors; they can also be integrated with artificial intelligence control.

• Mannequins are life-sized, human-like models of the entire body or a particular subset [96]. Many mannequins support various physiological functions, including detectable pulses and simulated breathing capabilities. Like standardized patients, they exhibit a high degree of physicality as a result of their realistic shapes and sizes, but they are generally static in appearance and unable to simulate certain visual symptoms. A task trainer represents only

a subset of the human body, such as an arm, which is sometimes all that is necessary for a particular training scenario; if desired, both a task trainer and another simulation technique can be used in a hybrid approach.

• Mixed-reality simulators are a special class of computer-based simulators [96] across a wide spectrum of techniques. Virtual reality patients can be presented in a fully three-dimensional virtual environment via head-mounted displays, affording a high sense of spatial presence. Additionally, spatial augmented reality supports the addition of virtual imagery, generally via projectors, to physical objects in the user’s environment, which could include virtual patients [13]. As they are computer-based, these simulations are all capable of displaying dynamic imagery, whether using head-mounted displays or projectors.

Each of the above patient simulators has benefits and disadvantages in terms of the variety of medical symptoms it can portray and its effectiveness in training. Physical-virtual patients [123] are a hybrid approach that aims to combine the valuable aspects of these simulation paradigms:

• Like mannequins, physical-virtual patients occupy real physical space, providing a sense of presence and physicality that can promote more compelling training scenarios and prompt more realistic behavior from trainees and learners.

• Being computer-based, physical-virtual patients support dynamic imagery through the use of augmented reality, allowing for the expression of complicated visual symptoms and a diversity of simulated patient appearances.

• Real humans can control physical-virtual patients, increasing the realism of speech and other behaviors.

Moreover, the increases in presence and physicality resulting from environmental awareness ex- hibited via touch are applicable to physical-virtual scenarios beyond patient simulation. General

virtual agents with sophisticated human-like capabilities are known as intelligent virtual agents (IVAs), described as

...intelligent digital interactive characters that can communicate with humans and other agents using natural human modalities such as facial expressions, speech, gestures, and movement. They are capable of real-time perception, cognition, emotion, and action that allow them to participate in dynamic social environments [10].

With such capabilities, IVAs are natural candidates for computer-based simulations of virtual hu- mans, including patients. However, IVA research has historically been limited to the visual and audio input/output domains [113]. Thus, among other such applications, one of our primary moti- vations for this research involves extending touch input to physical-virtual patient simulators and to IVAs in general, which is naturally handled by the presented methodology.