DE LA OFERTA ADMINISTRATIVA

All students who participated were pre- and post-tested. The mean scores are given in Table 5.6. The experimental group scored a mean of 5.9 out of a possible 12 for the pre-test. The control group scored a mean of 6 for the pre-test. Since the difference of mean scores is statistically insignificant we can conclude that the two groups have similar knowledge in ER modelling and are comparable.

Pre-test s. d. Post-test s. d. Gain score s. d.

K

ER

ER-Tutor 6.00 2.18 6.29 2.09 0.29 2.46

Table 5.6: Mean pre- and post-test scores of the experimental and the control group

The group of students who interacted with KERMIT scored 6.50 on average for the post-test, and

the group who interacted with the control system (ER-Tutor), scored 6.29. The experimental group

scored 0.64 higher in the post-test, whereas the control group gained 0.29. The t- test revealed that the difference between the gain scores is statistically insignificant. A statistically significant gain cannot be expected from such a short interactive session with the system. However, it is promising at least that the value of the mean gain score of KERMIT was higher than the mean gain score of the

control system. Moreover, the student’s performance did not degrade after interacting with the system, which confirms that the teaching system cannot have any negative impacts on ER modelling knowledge.

We computed the effect size and power, which are the two measures commonly used to determine the effects and validity of an experiment. Effect size is a standard method of comparing the results of one pedagogical experiment to another. The common method to calculate the effect size in the ITS community is to subtract the control group’s mean gains score from the experimental group’s mean gain score and divide by the standard deviation of the gain scores of the control group [Bloom, 1984]. This calculation yields (0.64 – 0.29) / 2.46 = 0.15. The resulting effect size is very small in comparison to an effect size of 0.63 published in [Albacete & VanLehn, 2000] and 0.66 published in [Mitrovic, et al., 2001a]. Both report experiments where students used

84

the system for a whole two-hour session. Better results on the effect size have been obtained in studies where interactions lasted for a whole semester or an academic year. Bloom [Bloom, 1984] reports an effect size of 2.0 for one-on-one human tutoring in replacement of classroom teaching and Anderson and co-workers [Anderson, et al., 1996] reports an effect size of 1.0 for a study that lasted for one semester. Considering these results, yielding an effect size of 0.15 with a study that lasted for only half an hour is quite promising.

Chin [Chin, 2001] published another method of calculating the effect size as the omega squared value (ω2_{). It gives the magnitude of the change in dependent variable values due to changes in the}

independent variables as a percentage of the total variability. ω2 _{is calculated using ω}2 _{= σ}

A2 / (σA2

+ σS/A2), where σA2 is the variance of the effects of varying the independent variable and σS/A2 is the

random variance among participants. According to this formula we get an effect size of 0.03, which is considered small in social sciences [Chin, 2001]. An effect size of 0.15 is considered large. The omega-squared value of the experiment further points out that the amount of time allocated for the participants to interact with the system was insufficient.

Power or sensitivity gives a measure of how easily the experiment can detect differences. Power is measured as the fraction of experiments that for the same design, the same number of participants and the same effect size would produce a given significance. In other words, power of 0.5 means that half of the repeated experiments would produce non-significant results. Chin [Chin, 2001] recommends that researchers should strive for a power of 0.8. We calculated the power of this experiment to find out how easy it is to detect differences in the pre- and post-test. The calculation yielded a power of 0.13 at a significance of 0.05, which is quite low. The low power value can be attributed to the low number of students, each group having only fourteen participants. Test A and B, used for the pre- and post-test, were designed to be of similar complexity. We compared the students’ performance on test A and B for the pre-test. Only the scores of the pre-test were taken into account to ensure the effects of interacting with different systems were discounted. The analysis revealed that students who sat test A had a mean score of 6.64 with a standard deviation of 1.7. On the other hand, the group who sat test B recorded a mean score of 5.21 with a standard deviation of 1.7. Although we assumed that the students would find test A and test B to be of a similar complexity, the students found test A slightly easier than test B. Kenneth Koedinger refers to this phenomenon of the experts making erroneous assumptions of the students as the ‘expert blind spot’ [Koedinger, 2001].

5.3.4 Discussion

Although the evaluation study was too short, the results from Study 1, conducted at Victoria University, has yielded some promising results. Students who used KERMIT displayed a slightly

higher gain score in comparison to the control group. Even though the differences are statistically insignificant, the findings are quite promising for such a short study. Most importantly, the pre- and post-test scores demonstrated that using KERMIT did not hamper students’ abilities in ER

modelling.

Students using KERMIT experienced a number of system crashes during their interaction. The

main reason for the crashes is the inadequate testing of the system running in the environment used for the study. There was little opportunity to test the systems at the computer laboratories of Victoria University where they were running on a Windows 2000 terminal server. Since the system was designed to run as a stand-alone program, a number of unexpected bugs emerged during the study. The control system (ER-Tutor), on the other hand, was more stable in comparison to KERMIT

as it was not required to evaluate the student’s solution. It would be fair to assume that the system crashes of KERMIT had a detrimental impact on the students’ perception of KERMIT.

One other factor that influenced the outcome was the students struggling to familiarise themselves with highlighting words in the problem text to specify each construct’s semantic meaning. A typical mistake made by the students was to add a new construct to their workspace, highlight a word from the problem text and rename the construct to have a different semantic meaning. In such cases KERMIT struggles to give useful hints, as it is designed to ignore the

construct’s name assigned by the user and only considers its tag associated with the highlighted area of the problem text (see Section 4.3.2 for details). This confusion can be minimised by preventing the users from renaming constructs, and by automatically naming constructs using the highlighted portion of the problem text. The change to the system would not be expected to hamper student’s learning, but would reduce the burden of having to type in the name of each construct. A rise in student performance could be expected as a result.

Students who used KERMIT were forced to complete each problem before moving on to the next

problem, whereas students in the control group chose a new problem at any instance they were dissatisfied with the current one. Students’ perception has been further affected by this variation. This is also a flaw of the experiment, since the group who used ER-Tutor were treated differently,

disqualifying them as a true control group.

Some students had difficulty in understanding the feedback messages presented by KERMIT. The

86

descriptive. Some messages delivered small hints that novice users struggled to understand. We plan to revise the feedback messages of the constraint base making them more descriptive.

Analysing the pre-test scores of both test A and B revealed that the students found test B harder in comparison to test A. The pre- and post-tests must be of a similar complexity to be comparable. Since the students found two tests to be of different complexities, the gain scores of both groups cannot be trusted to portray true knowledge gains.