01/01/2020 La revista Metro Ciencia es una publicación trimestral

Despite similar amounts of time on the material, students’ methods of going through the material were different, which can be seen in the box and whisker plot of the number of submissions students averaged for each question within the levels. This plot is shown in Figure 4.4. This method of representation was again chosen due to the skewedness of the data and the number of upper outliers. As before, the median is represented by the horizontal bar, which shows differences between the two groups. The Mastery group had a lower median number of submissions for every level, with the difference significant on levels 2 and 4 (p <0.01 via the Mann Whitney test, which was chosen because the distributions were not normal). This is unsurprising, since students in the homework group were not penalized (by getting a new set of questions) for wrong answers, and many resorted to trial and error. In an extreme example, one student had 93 attempts at a single question. When given feedback that their answer was off by a power of ten, students tended to try other iterations of their answer incrementally rather than going through their work and re-calculating. Trial and error was not a reasonable strategy for students in the Mastery group, as missing a question on the first submission meant that they were given a new set of questions, where their previous work was not relevant to finding the exact answer to the isomorphic new set. Some students in the Mastery group did “skip” the first version, submitting answers without spending time solving to go straight to solutions, but this is indicative of their ability to recognize themselves as novices and they were able to apply what they learned from the worked examples to different questions on the next set.

The differences in format delivery also informed some differences in student progression through the set or sets of problems. Figure 4.5 shows the average fraction of students correctly answering each question on their first attempt. Because students in the homework group were able to see feedback on questions before their first attempt on consecutive problems, they showed greater improvement question to question. For example, Questions 1 and 2 in Level 1 were similar; students in the Homework group did much better after they had received feedback and been able to work through Question 1 to correctness before attempting Question 2. The students in the Mastery group did not receive feedback until they had answered all questions in the set, and thus performed very similarly on the two similar questions. The same pattern is seen in Questions 1 and 2 in Level 4; Mastery students scored similarly on the two, but the Homework students showed improvement from the first question to the second.

In some questions (indicated by asterisks in Figure 4.5), the mastery students saw answer choices as multiple choice while the homework students entered their answers as free responses. It is possible that this was an advantage for mastery students; if they first calculated an answer that was not listed, they could try again. Investigating the content of the free response answers to attend to this difference gave increases in scores on four questions for the homework students (if we throw away any answers that are not listed as multiple choice options), but none that affected the general trends discussed. The largest gain was in question 2 of level 2, which increased to 58%, and other increases, which were much smaller, were in question 2 of level 3, and questions 1 and 2 of level 4.

For questions without format differences, such as questions 1, 3, and 4 in level 2, mastery students still significantly outperformed the homework students. Level 2 used many of the skills taught in Level 1, so their first attempt advantage suggests that the mastery students learned more from their work on the previous level. In level 4, questions 1 and 2 were very similar to the first question in Level 3, but homework students did not show improvement from the level 3 question into level 4, whereas the mastery students performed

Figure 4.4: Box and whisker plot of students’ average number of submissions per question within each level, based on their group. Horizontal black lines represent median values, which are within the shaded box representing the interquartile range (IQR=Q3-Q1). The whiskers represent either the highest/lowest values of time spent for that group, or 1.5 × the IQR beyond the edge of the IQR, in the case where there were outliers beyond that value. The outliers are not shown on the plot.

Figure 4.5: The fraction of students with the correct response on their first try on each question. Asterisks indicate that the questions had different modes of inputting answers; Mastery students always had multiple choice answers, but questions with asterisks were free response for Homework group students.

very well. Again, this suggests that the mastery students learned more from their work on Level 3, which they were able to apply to questions in Level 4, whereas the success by the homework students in Level 3 did not carry forward.

4.3.3

Posttest Performance

On the common written assessment, the improved performance of the mastery students persisted. Out of 30 points, the students in the mastery group scored an average of 16.0 ± 1.2 compared to the homework group’s average of 9.7 ± 1.3, which was statistically significant with p <0.005.2 _{A plot of the overall score and scores}

by question is given in Figure 4.6. The transfer between the treatment materials and the posttest varied from near to far transfer by question. The most similar question was Question 8 on the posttest, which was very similar to question 3 of level 4 in the treatment materials, and showed the largest gain for mastery students over the homework group. The question required students to integrate over two regions, one inside a conductor, and relied on skills that were trained throughout Levels 3 and 4. The mastery students also outperformed the homework group when combining expertise of electric fields in planar geometry and electric potential in spherical geometries to find electric potential differences in a planar geometry (questions 2, 4, and 6). Question 5, where the two groups performed the most similarly, asked students to work backwards

2_{The difference between the scores was significant whether using either grader’s scores or the average of the two, which is}

Figure 4.6: Outset: Mastery group students’ average score compared to homework group students’ score by question. Points above the line indicate that the mastery group outperformed the homework group. Inset: Posttest average scores for both matsery and homework groups, with control average score (no treatment) from the previous experiment included for reference.

from a potential difference in a planar geometry of two charged sheets to find the charge density of one unknown sheet. This required a sophisticated synthesis and transfer from the training materials.

4.4

Discussion

The experiment was designed to evaluate a mastery delivery system for learning materials compared to more traditional online homework. The students in the mastery group saw multiple versions of questions, as needed, coupled with delayed feedback for each iteration and followed with narrated animated solution videos, while students in the traditional homework group had unlimited tries on a single version, immediate feedback by question, and access to notes, colleagues, and online resources. On the written posttest, the mastery group signficantly outperformed the homework group, despite spending nearly identical time on the training exercises. They performed better on the near transfer question, and the questions which required synthesis of the skills during the treatment. The mastery delivery system takes advantage of formative assessment, multimedia principles, the worked example effect, and other learning principles which may account for students’ improved performance.

The students in the traditional homework group exhibited behaviors they believed would get them through the treatment as quickly as possible, often resorting to trial and error to find the answer. Because the

stakes were extremely low for submitting an incorrect answer, there was little incentive to check their work and students occasionally used incremental guessing to attempt to find the correct answer. The homework students had access to notes and online resources but rarely used them until they had exhausted their initial ideas or guesses. This may explain why students in the traditional homework group submitted significantly more attempts at the answers. In short succession of each other, they could transfer one question’s strategy to a similar question, but failed to do the same when the questions were spaced out and on different levels. In contrast, the mastery group worked more conscientiously in response to the higher stakes, double checking their work and using sense-making strategies to avoid the penalty of redoing an entire set of problems, observed anecdotally during the sessions. The students showed visible signs of stress when submitting answers to be graded, and likewise were vocal about disappointment or excitement after their sets were scored. The mastery students showed outward signs of investment in the mastery activities. This suggests that the higher stakes of the submissions increased student engagement and sense-making, which led to a larger learning gain from as the levels progressed, and higher performance on the posttest.

The changes to the experiment from the previous clinical trial had negative and positive consequences with respect to students’ frustration. Breaking the larger levels into smaller pieces gave students more opportunities to feel success, which they often celebrated aloud, but the increase of the mastery threshold from 85% to 100% slowed some students down on the final levels. We expected that breaking the material into smaller masterable chunks would help students work through the material faster, but both experiments had students averaging around 2 hours on the treatment materials. In the last levels, students were expected to keep track of many small steps and negative or positive signs, so giving them no leeway for error made it more difficult to master those problems. Some students complained of fatigue and frustration, as in the previous experiment. This is a consequence, in part, of the requirement that students complete the whole assignment in a single sitting, which would not necessarily be true for students doing homework for a course.

The purpose of this clinical trial, in part, was to see the suitability of mastery style online exercises for a course, as an improvement over the traditional online homework used in courses. The previous and current clinical trials gave evidence that the mastery delivery system was more effective, and the narrated animated worked solutions are effective correctives to use within the mastery delivery system. Students are able to master the materials with the support of the worked solution videos, as well as better carry their learning into near and far transfer problems. The results from these mastery clinical trials were encouraging and set the stage for its inclusion in an introductory course.