COMPARACIÓN ENTRE EQUIPOS NANOSTATION5 Y ANTENAS

Level of Education

We investigated the possible moderating effect of level of education by comparing pre- and post-test scores separating participants in higher general education (HAVO,

n = 22) from those in pre-university education (VWO,n = 11). By splitting into two

sub-samples the samples sizes are lower then 25, therefore, results should not be

read as hard proof but rather give an indication.

First, a Mann-Whitney U-test was conducted to compare the overall pre-test scores for HAVO and VWO students. There was no significant difference in the overall pre-test scores for HAVO (M dn = 12.5) and VWO (M dn = 13) students,

Z = −0.61, p > 0.54. Then, a Mann-Whitney U-test was conducted to compare the

overall post-test scores for HAVO and VWO students. Once more, there was no sig- nificant difference in the overall scores for HAVO (M dn= 3.5) and VWO (M dn = 6)

students,Z =−0.92, p >0.35. These results suggest that, initially, HAVO and VWO

students hold similar (mis)understandings of variables in imperative programming, and that the intervention was as effective for HAVO as for VWO students.

More detailed analyses revealed no differences between HAVO and VWO stu- dents in misconception scores, individual identified misconceptions, learning goal scores, or unanticipated behavioural mistakes for both pre- and post-test. With exception of learning goal 3, for which there was reported a significant difference

between HAVO (M dn = 0, M = 0.0) and VWO (M dn = 0, M = 0.18) students,

Z =−2.03, p <0.05. However, learning goal 3 scores were calculated from C1and S3 scores who have been rated respectively zero and four times in total. Although,

the samples are to small to base any conclusion, these results strengthen the pre- vious suggestion that the effectiveness of the intervention in addressing identified misconceptions was not dependent of or influenced by the level of education.

Noteworthy, Mann-Whitney U-tests did show significant differences in pre-test assumed prior knowledge scores between HAVO (M dn = 1, M = 1.82) and VWO

(M dn = 3, M = 3.82), Z = −2.66, p < 0.01, and in post-test assumed prior knowl-

edge scores between HAVO (M dn = 0, M = 0.77) and VWO (M dn = 3, M = 2.64),

Z = −2.86, p < 0.01. In other words, VWO students made significantly more mis-

takes indicating a lack of assumed prior knowledge then HAVO students in both the pre- and post-test. This difference between HAVO and VWO students is also reflected in Wilcoxon Signed-ranks test conducted on the split sample; indicating

that in the HAVO sample the post-test assumed prior knowledge scores were signif- icantly lower than the pre-test assumed prior knowledge scores,Z =−2.18, p <0.03,

while in the VWO sample the assumed prior knowledge scores did not significantly differ between the post- and pre-test, Z =−0.93, p > 0.35. Meaning that the HAVO

students initially showed less indications of a lack of assumed prior knowledge, and further improved their assumed prior knowledge after the intervention, than VWO students.

Chapter 6

Discussion

The results strongly indicate that the interactive video was effective in addressing identified misconceptions about assignment statements in variables of primitive type for secondary education students enrolled in a introductory programming course.

Although evaluation of the assessment tests was subject to an earlier report and outside the scope of this document, we wish to emphasise that there were some limitations towards these tests abilities to detect misconceptions. First, not all iden- tified misconceptions were detectable by a equal number of questions; the most extreme case, misconception S5, was uniquely detectable by only one question.

Moreover, incorrect responses were not seldomly observed in different questions than those designed to assess a certain misconception (see Table 5.1). Secondly, many predicted values mapped to certain identified misconceptions did not differ- entiate between two or more misconceptions, and sometimes between the correct answer (see Table 2.2). Lastly, a relatively large number of observed incorrect re- sponses were not mapped to identified misconceptions; in the post-test over half of the mistakes indicated a lack of assumed prior knowledge or could not be identi- fied (see Table 5.2). Due to these limitations, hindering detection of misconceptions held in the sample, it seems likely that not all misconceptions held by students have been effectively detected — both before and after watching the interactive video. Further, not all identified misconceptions may have been held by the students in our sample. The container analogy was not instructed to these students (who all re- ported no prior programming experience) and thus less likely to be hold. Therefore,

never detected misconception C1 and misconception S5 have been omitted from

this discussion, as is learning goal 6 which was planned being evaluated by mis- conception S5score only. In the remainder of this chapter we interpret the results

for the remaining identified misconceptions and learning goals in the light of these limitations.

6.1 Learning Effect of Interactive Video

Overall, students showed improved understanding of variables, providing less in- correct responses mapped to identified misconceptions or indicate lack of assumed prior knowledge after watching the video compared to before. Before an in depth discussion on the effectiveness of the intervention on specific learning goals and identified misconceptions, we first discuss some of the noteworthy findings for indi- vidual students and assessment questions.

Class of Errors In total students provided less incorrect responses in the post-test (Sum = 144) compared to the pre-test (Sum = 391). Of these, in the pre-test 247,

and in the post-test59, incorrect responses mapped to the predicted values indicat-

ing an identified misconception. This means that while the overall the number of incorrect responses decreased by 63%, those mapped to identified misconceptions

even decreased by 76%. This indicates that the video was highly effective in ad-

dressing identified misconceptions. However, the video was less effective in helping students to overcome unanticipated behavioural mistakes. In the pre-test a total of

27participants made62unanticipated mistakes, compared to39mistakes by22stu-

dents in the post-test. With a37%decrease students still showed to improve on their

unanticipated behavioural mistakes, but lesser than for those mapped to identified misconceptions. Also for incorrect responses related to assumed prior knowledge, with a nearly 45% decrease, the video showed helpful but not as effective as for

identified misconceptions. This is not surprisingly considering the fact that the video was designed to instruct learning goals explicitly addressing identified misconcep- tions, less so teaching assumed prior knowledge and not addressing unidentified misconceptions underlying unanticipated behavioural mistakes.

Individual Students All but one participant performed better on the post-test than the pre-test. Participant V0015JA provided six incorrect responses (mapping 2 to prior knowledgeEand 4 to misconceptionM2) in the pre-test and seven (mapping 3

to prior knowledgeE, 3 to misconceptionM2and 1 to misconceptionS4) in the post-

test. These small numbers do not inform any suggestions on the relation between the intervention and deteriorating performance; the student might as well have been tired or distracted. Further, while in the pre-test none of the participants provide all correct responses, in the post-test six participants did so. Additionally, 13students

provided five or less incorrect responses in the post-test. Given that some of these students provided as much as 19incorrect responses in the pre-test we can safely

6.1. LEARNING EFFECT OF INTERACTIVEVIDEO 39 Individual Questions In total, 24 out of 28 questions were answered correctly more frequently in the post-test than the pre-test. One question, n2, was answered correctly by all students in both the pre- and post-test. The remaining three ques- tions received one or two more incorrect responses in the post-test than the pre-test. Questioncwas answered incorrectly in the post-test by one participant,H004BB, who provided the predicted value for misconceptionH1. Also in the pre-test, this partici-

pant provided once the predicted value for misconceptionH1, but then for question

d. Question f2 was answered incorrectly by none of the participants in the pre-test

and by two in the post-test; both incorrect responses could not be identified as de- tecting an identified misconception or lack of assumed prior knowledge. Questionm2 received one (unidentified) incorrect response in the pre-test, while in the post-test two participants provided the predicted value mapped to misconception S1. Inter-

estingly, for both question post f2 and post m2, one of the two participants who provided an incorrect response was participant H004BB. This participant provided

0 for all three questions. Alternatively, to showing to hold misconception H1 form

questionpost c, or misconceptionS1form questionpost m2, s/he may have chosen

the Visual Basic default value for Integers whenever unable to give the correct value. The fact that variables have a default value, assumed prior knowledge prior knowl- edgeE, appeared not fully understood by the participant in the pre-test, while in the

post-test s/he did not provide any incorrect responses indicating a lack of assumed prior knowledge.