Plan Técnico

In the following, the students’ perception questionnaire is presented, as well as the hypothesis that each respective question is aimed at verifying. In the design of the questions, we have taken care not to insert any bias:

1. I believe that, generally speaking, MOT 2.0 changes the learning outcome.(H1)

 Increases learning outcome (learning the selected lesson).

 Decreases learning outcome (learning the selected lesson).

 No influence on learning outcome.

2. I believe that, generally speaking, MOT 2.0 changes the learning outcome.(H2)

 Easier.

 More difficult.

3. I believe that, generally speaking, MOT 2.0 changes the learning outcome.(H3)

 Better.

 Worse.

 Neither better nor worse.

4. I believe that, generally speaking, the interaction with the system is:(H4)

 Easy to learn.

 Hard to learn.

 Neither easy nor hard.

5. I believe that, generally speaking, the interaction with the system is:(H5)

 Easy to remember.

 Hard to remember.

 Neither easy nor hard.

The answers were further mapped, for numerical analysis, over the integer values of {-1, 0, 1}, with ‘-1’ representing the negative answer (e.g., for question Q1, ‘Decreases learning outcome’), ‘1’ representing the positive answer (e.g., for question Q2, ‘Easier’) and ‘0’ representing the neutral answer (e.g., for question Q3, ‘Neither better nor worse’). The assumption underlying this mapping is that there is an implied monotonicity in the answer range, as well as a symmetric, equidistant relation between the positive and the negative answer. This is a relatively strong assumption, which we consider compatible with the type of answers we have selected, and which is in regular use in such questionnaires for learning systems. Out of the 16 students who took the post-test, 94% of the students also took the optional questionnaire as described above, see Figure 41.

The questionnaire results represent the students’subjectiveperception about the MOT 2.0 system (as opposed to the test results, which represent the objective measure of the learning outcome). Therefore, 93% of the students declared that the MOT 2.0 system changes the learning outcome, and only one student declared that it has no influence on the learning outcome. Students were also asked to comment on their answers, but not all of them did, as comments were not obligatory. The only two students commenting (which we further anonymize by calling them students A and B) added the following comments: Student A: “I like the way i can view the structure and see which is a subpart of which.”; Student B: “much faster”. Clearly the two students perceive that the system has helped them, although in different ways.

Overall, a statistically significant majority felt that the system helped in learning, regardless of the group to which they belonged, thus confirming hypothesis H1. The next question, with a similar majority of opinion (87% of the students) concerns user satisfaction, which is increased by using the system, thus validating hypothesis H2. For question 3, 80% of the students believe that MOT 2.0 (third prototype) has increased their learning outcome. Therefore, the statistically significant positive result validates hypothesis H3. For question 4, corresponding to hypothesis H4, 87% of the students believe the interaction with the system is easy to learn. Overall, hypothesis H4 is thus confirmed. For question 5, fewer students (76% of the students), but still a statistically significant majority, believed that the system is easy to remember. Therefore, the statistically significant positive result validates hypothesis H5. Overall, all five hypotheses were confirmed, based on statistically significant results, as shown below:

Table 12 One-Sample T: Q1, Q2, Q3, Q4, Q5

Questions N Mean StDev SE Mean 95% CI T P

Q1 15 0.93 0.26 0.07 (0.79 to 1.08) 14 0.00 Q2 15 0.87 0.35 0.09 (0.67 to 1.06) 9.539 0.00 Q3 15 0.8 0.41 0.11 (0.57 to 1.03) 7.483 0.00 Q4 15 0.87 0.35 0.09 (0.67 to 1.06) 9.539 0.00 Q5 15 0.87 0.35 0.09 (0.67 to 1.06) 9.539 0.00

Next, we attempted to evaluate the questionnaire results from the point of view of the group that a respective student has been placed in. Thus, for each group (G1, G2), we re- examined the answers to the questions (Q1 – Q5) and thus the rejection/confirmation of the respective hypotheses (H1-H5). The results of this analysis are shown below. Thus, a result of P<0.05 for Qi_Gj confirms that for group Gj hypothesis Hi remains valid, as for all questions Qi the results were on average positive, and close to 1 (see the Mean column in Table 13).

Table 13 One-Sample T: Q1_G1, Q2_G1, Q3_G1, Q4_G1, Q5_G1, ... Q5_G3

Question H N Mean StDev SE Mean 95% CI T P

Q1_G1 H1 6 0.83 0.41 0.17 (0.40 to 1.26) 5 0.00

Q2_G1 H2 Cannot compute T-test. All values are the same (value is 1).

Q3_G1 H3 6 0.5 0.55 0.22 (-0.07 to

1.07)

2.2361 0.07

Q4_G1 H4 6 0.83 0.41 0.17 (0.40 to 1.26) 5 0.00

Q5_G1 H5 6 0.83 0.41 0.17 (0.40 to 1.26) 5 0.00

Q1_G2 H1 Cannot compute T-test. All values are the same (value is 1).

Q2_G2 H2 8 0.78 0.44 0.15 (0.44 to 1.12) 5.2915 0.00

Q3_G2 H3 Cannot compute T-test. All values are the same (value is 1).

Q4_G2 H4 8 0.89 0.33 0.11 (0.63 to 1.15) 8 0.00

Q5_G2 H5 8 0.89 0.33 0.11 (0.63 to 1.15) 8 0.00

As can be seen in the table, all hypotheses H1 to H5 were confirmed for group G2, which is the group with all the added adaptive content and recommended peers. However, H3 (about learner satisfaction) could not be confirmed for group G1 in the 95% interval (only in the 90% interval). The rest of the hypotheses were confirmed for group G1. Looking at the differences in the opinion between the two groups, group G2 slightly outperforms group G1 (the control group) on hypothesis H1 (about increased learning outcome; with a difference of 0.17), clearly on H3 (about increased learning satisfaction; with a difference of 0.5), slightly on H4 (about ease of learning and using; with a difference of 0.06), and similarly slightly on H5 (about ease of remembering; with a difference of 0.06). The only

hypothesis where group G1 outperforms group G2 is H2 (about decreasing learning effort; with a difference of 0.22). This may indicate that recommendations have helped in every way, but may be perceived as added effort on the part of the student.

Further on, we have examined the time the learners have spent on the various activities. On average, the students in the control group G1 spent less time (seven minutes less) studying than the ones in group G2. As the studying time was up to them (they all had to finish within two hours, but other than that, no fixed time was set), this could indicate an increase in interest in studying for the students who benefited from recommended content and recommended peers.