Físico: Que comprende dos aspectos:

Most studies detecting affective and cognitive states within serious games interaction sessions were limited to a small number of traditional emotions (see Table 4.2) which do not cover all aspects related to the three dimensions learning, gaming and using in addition to the context dimension, which are, as discussed in this chapter, crucial for understanding learner experience in serious games [192]. A more general approach was thus needed to include a reasonable set of states worth capturing during a serious games evaluation session in the field.

This should consider aspects related to affect, cognition, behavior and context which are possible to encounter during an educational game session and related to the dimensions: learning, gaming, using

and context. To do this, it will need to take into account measures originally defined for pure learning environments, others defined for games as well as measures defined for user experience evaluation. The focus of the study is not how to recognize these features using automated methods nor which modalities are best suitable for capturing the data but rather define which aspects need to be considered to gain a better understanding of the full experience.

As event logs typically capture interactions with the game environment, it needs in many cases to be enriched by user reactions and other context data to be more effectively analyzed. Consider a user being inactive for a remarkable time period inside a serious gaming session. The log data will capture the time elapsed before an interaction occurs but more data will be needed to disambiguate the reasons behind this behavior to properly determine the resulting design decision.

The reasons for such a response delay can lie on the learning dimension if the user is found to be reflecting on the answer to a question or on the gaming dimension if the user is reflecting on the strategy to make the next game move, for example. Furthermore, the problem might be related to usability if the user is found to be confused on how to proceed with the navigation, for example.

However, the reason might also be exterior to the whole gaming experience, as it might lie in the context: the user might have been distracted by an event in the environment which happens frequently in real-life situations or when testing in the wild. All these factors need to be taken into account for an ideal evaluation. For this, we will assume an observer watching the testing session and being able to take note of states related to these dimensions which can have influence on the experience at the moment they occur. This would be the ideal situation for capturing all relevant data which, in real testing scenarios, however, will naturally suffer from many limitations. Nevertheless, it can serve as the ultimate goal of recording the experience, whether doing this manually or automatically. Table 4.2 presents parameters of Affect, Cognition, Context and Behavior considered in different models investigating data in learning environments.

After reviewing the different parameters considered in literature, it was found that the best approach would be to carry out a study to determine which of the factors present in literature will be elicited during playtesting sessions of Serious Games carried out in the field. This would help decide which of the above listed models more precisely fits the evaluation of (mobile) educational games in natural environments on each of the three categories: Affective and Cognitive States, Context and Behavior.

Based on the results which will be discussed in the Evaluation chapter, the chosen states are depicted in Figure 4.8.

Table 4.2:Evaluation States Identified in Literature to Occur in Interaction Environments.

Citation Environment Affective and Cognitive Context Behavior

[194] Online learning frustration, anxiety, shame, excitement, pride [51, 53]

Educational games joy, regret, admiration, reproach, pride, shame

[19, 18] Intelligent Tutoring Systems

boredom, confusion, delight, engaged concentration, frustration, surprise, neutral

on-task, on-task conversation, off-task Conversation, off-task solitary behavior, inactivity, gaming the system

[79] Game-Based Learning interest, enjoyment, task involvement, confidence [234] E-learning interest, engagement, confusion, frustration, boredom, hopefulness [201, 203, 204] Academic settings

enjoyment, hope, pride, relief, anger, anxiety, shame, hopelessness, boredom

[91, 52]

Educational games

boredom, shame, frustration, confusion, disappointment, surprise, neutral, curiosity, engaged concentration, delight, excitement, confidence, pride

[29] Mobile learning

location, time, weather, temperature, noise, lighting, day, movement, device capabilities

[284] Mobile learning motivation noise, busyness of the environment, temperature [85] Mobile Learning

people, hardware and software capabilities, place, time, noise, illumination

[67] Mobile learning

enthusiasm, boredom, happiness, sadness , satisfaction, calmness, anger, anxiety, frustration, fear, confusion, hope, pessimism, expectancy , astonishment, sympathy, disgust, hate, pride, shame

time, location, terrain, weather, neighbors mobility, device capabilities

4.6 Conclusion

The need for understanding serious play experience has led to more and more research being conducted to use multimodal methods in the evaluation of serious games. However, available theoretical evaluation frameworks for Serious Games do not consider the use of multimodal data despite the increasing number of studies using them and despite their many benefits discussed in this chapter. After an overview over available frameworks for the evaluation of Serious Games was given in the last chapter, the Reasons and Responses Model was presented, with the aim of filling a gap in theoretical foundations for Serious Games evaluation by focusing on the value of adding multimodal data to event logs. First, dimensions of evaluation aspects were defined based on a literature review. Then it was examined which role multimodal data can play in measuring these aspects, specifically in determining reasons behind users’ logged gameplay actions and their responses to game events using the proposed model. Finally, practical examples of using this model for combining logging with multimodal data in evaluation were discussed. Establishing this framework, which was also published in [238, 246] will be applied in developing a

multimodal Serious Game evaluation tool as a proof-of-concept which will be the next step in the current research. It is answering parts of Research Questions 1-3 as it defines parameters for a serious games evaluation (Research Question 1), defines a hypothesis of why and when multimodal data is needed to interpret log events in Serious Games Evaluation (Research Question 2) and defines different types of multimodal data needed in evaluations and when they can be used (Research Question 3).

5 StoryPlay Multimodal: A Research Platform for the Mutimodal Evaluation of Serious Games

This chapter presents the design and development of StoryPlay Multimodal, a mobile multimodal ana- lytics platform for the evaluation of Serious Games. It is intended to serve researchers, teachers and educational game developers as a means to assess their Serious Game Design. This is done by captur- ing, pre-processing, synchronizing and visualizing multimodal serious games analytics and mobile sensor data from playtesting sessions. By linking log data with multimodal data, it is possible to uncover rela- tions between design elements, gameplay interactions, context parameters and affective and cognitive states. This is crucial for gaining full insight into a session, even if not present with the player at the same location. After discussing design requirements, the architecture of the software, the different modules, additional features, implementation challenges and solutions are presented. Parts of this chapter are published in [237, 239].