Una clasificación del diario de información general

The data collected in the suggested experiment are purely quantitative in nature and thus re- quire quantitative analysis. We hypothesize that the various forms of gamification employed could increase both the amount of user participation as well as its quality. The amount of user participation is measured by the amount of events triggered by each individual user within the experimental time frame.15 _{The quality of user participation is difficult to assess}

automatically. We therefore suggest the use of the teachers’ evaluation of student compe- tencies as a measurement for the quality of submissions. This seems appropriate, as the intended use of the system is that the students submit material through DAKORA as proof of their competence. Submission quality is a combination of the amount of competencies gained and their level. We suggest the hypotheses shown in table 5.2 for testing in the ex- periment. One should test whether each gamification element separately causes an increase in quantity or quality of submissions compared to the control group as well as compared to each other and to a combination of both.

Table 5.2: Hypotheses to be tested in the experiment. Each hypothesis has the fol-

lowing form: Awarding [condition] increases user participation [outcome] compared to the [comparison] group.

# condition outcome comparison

A1 badges (A) quantity control (D)

A2 badges (A) quality control (D)

B1 points (B) quantity control (D)

B2 points (B) quality control (D)

B3 points (B) quantity badges (A)

B4 points (B) quality badges (A)

C1 both (C) quantity control (D)

C2 both (C) quality control (D)

C3 both (C) quantity badges (A)

C4 both (C) quality badges (A)

C5 both (C) quantity points (B)

C6 both (C) quality points (B)

15_{The analysis described here does not take changes in behavior over time into account as it is designed}

for the minimum time frame of a single semester and the minimum number of particiating schools. Under these conditions, data will not be robust enough for a proper analysis of changes in system usage over time as individual factors are highly likley to skew those results. Under different circumstances, especially a much higher number of participating schools, a time series anaylsis would become very interesting. The software is already set up for such analysis, as all data are recorded with timestamps.

The chosen measurements should allow relatively simple statistical testing of the afore- mentioned hypotheses. With an unknown distribution of data, we suggest using the Mann- Whitney U test (see Mann & Whitney, 1947), a non-parametric test of the equivalence of the distribution of two independent sets of data, as employed for example by (Denny, 2013). If a significant difference is found, a simple calculation of the median of both sets of data should suffice to test for the hypothesized increase of quality or quantity. Depending on which of the 12 hypotheses above are confirmed, this will help us understand whether the employed gamification elements increase desired user behavior compared to the control group (hy- potheses A1 and B1), which one has a higher impact on that behavior (B3), and whether combination of the two leads to an even higher impact (C1, C3, C5). The corresponding even-numbered hypothesis deal with the quality of student contributions depending on the gamification elements employed, as measured by teacher ratings. For a pure measurement of the impact of gamification, all measures of quality could be dropped, as discussed in sub- section 5.2.3. From a practical point of view, however, it is interesting to see whether adding gamification elements to an LMS can actually help increase learning16 _{(whether mediated by}

behavioral factors as suggested by Landers (2014) or not) and the system already provides basic data for submission quality. Past results would indicate that we are likely to see an in- crease in quantity rather than quality of submissions with these basic gamification elements, but the reverse is theoretically possible. In the unlikely case that the only significant effects are those in quality of submissions, a focus on the odd-numbered hypotheses would hide that fact. Similarly, a potential decrease in submission quality caused by students merely trying to gain rewards through submission quantity would be hidden by such a design.

The analysis suggested here is purely summative in nature and does not take the time distribution of activities into account. Given a relatively short experimental time frame of one semester of study and the minimum number of participating schools, other analyses are unlikely to provide significant results as they would introduce additional biases. If one is able to increase either the number of participating schools or the time frame that data

16_{Again, actual measurements of learning outcomes would be much more complicated. An increase in}

submission quality could also be seen an indicator of another behavioral factor, such as the amount of time spent on producing high quality submissions. Nevertheless, better teacher ratings of student submissions would suggest that learning may have been increased.

is collected in — or ideally both — large numbers should help to average out individual differences in timing in different schools and classes. Snow et al. (2015) describe three types of methods for dynamic analysis of log data in educational games — random walks, entropy analysis, and hurst exponents — that could be adapted for the analysis of our log data as well. Access to a sufficiently large number of participants would also help in answering the question of different types of users of gamified systems that we have discussed in section 3.3. Cornforth & Adam (2015, p.152) used cluster analysis to find different types of profiles in the game Minecraft, claiming that such analysis allows “(1) to systematically investigate how learning depends on individual characteristics and (2) to adjust the learning approach accordingly”. In the case of gamification of LMSs, an understanding of how different types of users react to different gamification elements could be very helpful.

Following these more theoretical preparations of our proposed experiment, the following two sections deal with the practical implementation thereof: The next section describes the application we developed, followed by a description of the issues we encountered with field access in education.