La teoría del framing

Previous empirical studies of gamification have measured a large variety of different out- comes. These measures can be roughly distinguished into two categories — measuring changes in behavior and measuring target outcomes. We have discussed this in more detail in subsection 2.2.1 for gamification in education, describing the model of gamified learning by Landers (2014) in which learning was the target outcome, influenced (among others) by gamification elements. Learning is a common target outcome in education, but other gami- fied systems are targeted at other outcomes — such as increased user fitness in a gamified sports application. While measuring the target outcome is certainly the most useful mea- surement when creating a system aimed at that outcome (c.f. Mayer, 2014, for a discussion of measurement in game-based learning), it is less useful when attempting to measure the effectiveness of gamification (or certain gamification elements).

Mayer (2014, p.37ff) takes a strong stance on appropriate measures in the research of games for learning.12 _{To him, academic learning outcome is the most important outcome}

of employing an educational game Mayer (2014, p.38). His arguments are strong and he describes an issue that can also be found in gamification research in many cases. There is a strong reliance on subjective, self-reported measures of factors such as enjoyment or motivation in both game and gamification research that does not help answer the question of whether the object of the research is actually effective at reaching its goal. That said, Mayer’s strict focus on learning outcomes can be problematic. For one, education has more goals than just academic learning. But even if we focus strictly on academic learning, other

11_{Holidays and exam periods for example are likely to have an impact on LMS usage.}

12_{While Mayer writes about games for learning instead of gamification, we consider his work to be pertinent}

measures can certainly be helpful and interesting. One reason for this is the difficulty of measuring learning outcome appropriately. For example, Liu et al. (2012) have shown that motivational conditions have a substantial effect on performance evaluations. Worse, the validity of many instruments for assessing learning outcomes is questionable at best, especially in the case of low-stakes assessment (see e.g. Wise & DeMars, 2005), such as a post-intervention questionnaire. Furthermore, game-based or gamified learning approaches have a tendency to foster practical learning as opposed to the improvement of conceptual knowledge (see e.g. de-Marcos et al., 2016). While Mayer is certainly correct that learning outcomes are a primary objective in games for learning, their assessment is difficult and any chosen form of assessment has the potential to hide benefits or to exaggerate them. One should therefore certainly assess games for learning for their learning outcomes, but be careful with construct validity and include additional measurements as well.

This research, however, is not so much concerned with game-based learning but rather with the gamification of parts of the learning process. From a purely scientific perspective, the main goal of this study is to evaluate the effectiveness of various approaches to gam- ification and learning is just one possible context of many. A focus on learning outcomes would not only assume that these are the primary goal of gamification in an educational context, but also remove any possibility for a transfer of results to other domains.

In his theory of gamified learning, Landers (2014) proposes that the effect of gamifica- tion on learning is less direct than that of game based learning anyway. Whereas in the context of game based learning13 _{instructional content and game characteristics combine}

to form the game cycle that produces learning outcomes, the impact of game character- istics on learning outcomes is only indirect in gamification. According to Landers, game characteristics mediate behavior/attitude which, in turn, mediates learning outcomes and moderates the mediation of learning outcomes by instructional content. This assumes that there are behaviors that can impact learning outcomes and that these behaviors in turn can be influenced through game characteristics. Even if learning outcomes are to be con- sidered the primary objective of gamification in learning, any measurement thereof would only indirectly measure the impact of gamification as such. Gamification may, for example,

Game Characteristics (D)

Instructional Content (A)

Behavior/Attitude (C)

Learning Outcomes (B)

Figure 5.3: Landers’ theory of gamified learning, after (Landers, 2014). Behavior and atti-

tude serve as both a moderating and a mediating intermediary between game characteristics and learning outcomes.

be successful in promoting a certain behavior but that behavior may then not result in an actual improvement of learning outcomes.

With a focus on gamification, and not learning, it is therefore crucial to measure not learning outcomes but changes in behavior. This avoids the pitfalls of learning outcome measurement and at the same time provides more direct information about the effectiveness of gamification. We do not know if causing students to increase their activity in the DAKORA application will cause increased learning, that is for others to find out.14 _{We concern}

ourselves with the question, whether gamification can increase said activity and whether different gamification elements differ in that effect. Data collection in the experiment therefore needs to focus on activity data, concentrated on those activities that are actually targeted by the gamification intervention. Other behavioral data could certainly enhance the results of the experiments, such as self reports of intrinsic motivation (using, for example, the intrinsic motivation inventory (see e.g. Ryan et al., 1991)) or assessments of the quality of student submissions, as employed by Denny (2013). Either would greatly increase the difficulty of a large-scale experiment, however. Any non-automatic data collection (such as questionnaires that students have to fill out) must be considered an intrusion into the natural use of the system, skewing results and reducing the willingness to participate. The large-scale analysis of the quality of student contributions would require a high amount of expertise

14_{For example, Mabed & Köhler (2012), in a study of the LMS OLAT, could not find a correlation between}

LMS usage and academic achievement or attitude towards learning, citing, among other factors, the content of the course as a potential reason. When studying the impact of LMSs from a task-technology fit perspective, McGill & Klobas (2009) found a moderate effect on students’ perceived learning but only a weak impact on student’s grades.

and manpower, further complicated by different levels of education. We suggest that a large- scale quantitative experimental setup such as ours should eschew such measures. Results could be complemented with additional, small-scale studies focusing on qualitative aspects. Although rarely found in the analysis of gamification and serious games, there are other outcomes that are worth evaluating. Conati & Gutica (2016) analyzed the emotions of players of an educational mathematics game as coded by expert observers. The authors found many occurrences of engaged concentration and confidence, followed by confusion. They point out that they found only few occurrences of pride, shame, and curiosity even though those are supposedly “important for learning in educational research” (Conati & Gutica, 2016, p.29). This could be seen as a confirmation of Mayer’s claim that measuring learning outcomes is more helpful than measuring intermediate outcomes. On the other hand, an understanding of the emotions caused by games or game elements may give us more immediate information about their effect than a raw measure of learning outcomes.

All data collection for the purposes of our proposed experiment can be done through our plug-in for Moodle. Every action that has the potential to lead to either the awarding of points or badges is stored in a database table (gamification_log). Each entry consists of the type of action taken, the id of the user that took it, the date and time that it was registered at, and a data structure that contains additional information depending on the action.

Additionally, all data generated by the Moodle plug-in through its usage is available for evaluation as well. Database tables of relevance here are the one that stores user badges (gamification_badge) as well as the one that stores the points for each user (gamifi- cation_points). In the case of Baden-Württemberg, each school uses its own database with Moodle. All data is delivered on a per-school basis and can easily be attributed to each experimental group. All data stored is pseudonymized through the user id, which is not publicly available (according to requirement 9 below). While Moodle’s database contains information that allows linking user ids to natural persons, the database tables used for evaluation here do not and they are the only ones that need to be made available to the researchers, alleviating privacy concerns. The following section provides a suggestion for the analysis of the quantitative data that our experimental setup provides.