different kinds of data. Facial electromyography (measurement of the movement of the muscles in the face) and facial expressions have been used as an indicator of positive and negative affect (Partala & Surakka, 2004), dimensions of emotion (Gilroy, Cavazza, & Vervondel, 2011), and to improve systems in terms by inferring relevance (Arapakis, Konstas, & Jose, 2009; Arapakis, Konstas, Jose & Kompatsiaris, 2009; Arapakis,
Athanasakos, & Jose, 2010) and recommendation (Arapakis, Moshfeghi, Joho, Ren, Hannah, & Jose, 2009). Myography has also been used to measure hand gestures (Saponas, Tan, Morris & Balakrishnan, 2008), along with mouse movements and pressure-sensitive keyboards as indicators of stress (Epp, Lippold & Mandryk, 2011; Sun, Paredes, & Canny, 2014; Hernandez, Paredes, Roseway & Czerwinski, 2014). Posture-sensitive chairs have been used to measure body posture as an indicator of emotional state (De Silva, Kleinsmith, & Bianchi-Berthouze, 2005). Pupil diameter and eye movement have been measured using eye-trackers or similar devices as an indicator of emotional state (Ren, Barreto, Gao, & Adjouadi, 2013; Cole, Gwizdka, & Belkin, 2011). Electroencephalography (EEG), the measurement of electrical brain activity, has been measured using sensors as an indicator of preference (Ellick, Mirza-Babei, Wood, Smith, & Nacke, 2013), emotional response
84
Skin conductance has been used in many different studies as indicators of slightly different phenomena. Galvanic skin response has been used as an indicator of cognitive load (Nourbakhsh, Wang, Chen, & Calvo, 2012; Solovey, Zec, Garcia Perez, Reimer, & Mehler, 2014), responses to interface changes (Pan, Chang, Himmetoglu, Moon, Hazelton, MacLean, & Croft, 2011), and stress (Mooney, Scully, Jones, & Smeaton, 2006). Heart rate monitoring is usually done via monitors similar to those used in health studies. Some studies have monitored heart rate directly by monitoring heartbeats (Anttonnen & Surakka, 2005;
Magielse & Markopoulous, 2009). Others have done this by extracting heartbeat from ECG signal (Cai, Liu, & Hao, 2009). Still other researchers have created their own wearable sensors to capture these same types of data (Fletcher et al., 2010) in addition to affectively intelligent interfaces (McDuff, Karlson, Kapoor, Roseway, & Czerwinski, 2012).
To better illuminate what relationship emotions have to these attributes, researchers in this area have combined physiological, behavioral, and other affective signals for feature extraction via machine learning, and self-reports of emotions (Arapakis et al., 2009). Affective signals have been used to improve systems by attempting to predict relevance as well as offer recommendations. Arapakis, Jose, and Gray (2008) linked facial expressions with emotions experienced during search tasks. Participants were asked to assess the difficulty, complexity, and ambiguity of three search tasks, as well as to report levels of difficulty, interest, and fatigue experienced. Using these subjective reports in combination with facial expression analysis, the researchers were able to link unpleasant emotions such as irritation and anxiety to perceptions of difficulty and feelings of fatigue experienced during a task. Facial expression analysis also proved useful for prediction.
85
Arapakis, Konstas, and Jose (2009) combined physiological measures and facial expression analysis with machine learning to predict which documents or video snippets would be relevant. They extracted facial expressions, heart rate, galvanic skin response, and temperature, and asked participants to view documents and videos for four search tasks and mark whether they were topically relevant or not. The authors found their models performed better for video content, suggesting that audio-visual stimuli is more emotionally-laden than text, but they acknowledged that stronger emotional reactions (based on facial and
physiological data) could have been elicited by the content itself, rather than the fact the participant marked the item as relevant. Later, Arapakis, Athanasakos and Jose (2010) compared personalized versus general affective models in terms of their ability to successfully predict topical relevance, and found personalized affective models overall successfully predicted relevance significantly better than general models.
Arapakis, Konstas, and Jose (2009) identified a problem with physiological data; it is often noisy, as signals can be elicited easily but are often difficult to interpret. This is where self-reporting of emotions can be helpful; interpretation can be elicited from the user (though this too can be problematic at times). This study also demonstrated a problem with using a machine learning approach to build a model with physiological data: since the data is noisy, it is difficult to extract distinct features that perform well. Most of the features extracted using the two different machine learning approaches (K-Nearest Neighbor and Support Vector Machine) did not exceed the baseline. Moshfeghi and Jose (2013) successfully combined facial expression, heart rate, galvanic skin response, and EEG signals with dwell time (a behavioral signal) to predict relevance. Again, using a machine-learning approach for
86
feature classification, they found that behavioral and physiological data could act as reliable signals for relevance.
Niu, Zhao, Zhu and Li (2013) took a novel approach to video recommendation by identifying the emotional state of videos. They used automated computing of the affective signal (meaning the overall emotion or emotional tone) of the video based on attributes such as frame rate change, audio pitch, and motion. These attributes were used to cluster videos with similar affective signal, and could be used effectively in a recommendation system. Essentially, if a system detected happiness in a participant, then it would recommend a video with a happy affective signal. The research in this area is progressing towards systems that are continually able to monitor a participant’s affective state and tailor their experience accordingly. Given that the work done in this domain is heavily based on visual stimuli, it is possible that affective signals would not be very useful in text-based recommendation systems.
Identifying emotional states can help pinpoint moments of stress. Lazarus and Folkman (1984) define psychological stress in terms of a coping response to difficult situations that produce negative emotions. However, though emotions have received some treatment in the information retrieval literature, the stress response has not. To understand stress fully as a phenomenon, linking self-report data to physiological measurements can give us a complete picture both of what the person is able to self-report as well as the stress they are unable to self-report. This essentially serves as confirmation that a stressful stimulus was present. Physiological measurement is not without its problems, however. Though the signals and experimental constructions are different, what is clear from the work done in this area is that the particular signal must be chosen carefully and appropriately for the context in which
87
it is studied. The data is still extremely susceptible to noise, as a cough or sneeze can produce false (or stimulus-independent) changes in both skin conductance and heart rate. The key to measurement of this kind effectively is testing and defining a stimulus and creating ideal conditions to measure it, and it may serve as a powerful indicator of stress. There is also the normalization factor; data that has not been normalized often shows unclear trends and is non-significant as compared with normalized data, which can show clear stimulus-linked patterns.
2.7.4. Problems and Recommendations for Measurement. Physiological