This section describes behavioural changes which were fostered with Backstage 2 / Projects. First, influences of analytics-based nudging (described in Section 4.3) are reported, then a general change in the use of the platform between two courses is described.
The influence of nudging. The learning and teaching format “Analytics-based Nudging” encompasses the provision of predictions of a student’s examination fitness (see Section 5.2) and risk of skipping homework (Section 5.1) to nudge students to deliver (more) homework. A first evaluation of this learning and teaching format was conducted in a course on theoretical computer science (TCS-2018). The results of this evaluation are published in [106]. In this course, in which personal analytics were provided in an online dashboard on the platform, students skipped fewer homework submissions than in the three previous course venues, as seen in Figure 6.4. While the homework skipping rate of 2018 was significantly lower to the skipping rates of 2015 and 2017, it was not significantly lower than in 2016. A second evaluation was performed in a course on logic and discrete mathematics (LDM- 2019). For this evaluation, Backstage 2 / Projects sent personalized analytics reports via email to the course’s participants, as well as providing an analytics dashboard (both are described in Section 4.3). Yet, despite the analytics-based nudging being more prominent than in the previous evaluation, no significant decrease in homework skipping was found: Skipping rates changed from 96.8% of all homework being skipped to 96.4%.
Both evaluated courses differed not only in the implemented nudging but also in taught subjects, teaching staff and other details. Furthermore, only in the TCS courses, bonus points for delivered homework were awarded, and students reported missing incentives as a cause of not submitting homework (reported in Section 4.3). This might be a major cause for the large difference in homework skipping rates between the TCS courses (between 61% and 74%) and the LDM courses (96.8% and 96.4%).
A culture of dialogue. In the following, data from LDM-2018 and LDM-2019 are evaluated. Between these course venues, the software remained fairly similar, with the only changes being the implementation of the afore-mentioned analytics reports, and an improved com-
commented feedback mean HFD length commented exercises mean ED length LDM-2018 1% 2 29% 2.75 LDM-2019 4% 2.4 9% 1.75
Table 6.1: Comparison of discussion lengths in LDM-2018 and LDM-2019, “commented feedback” refers to the frequency in which students commented on feedback they received from a teacher, “HFD length” refers to the total numbers of comments in a homework feedback dialogue, and “ED length” refers to the total numbers of comments in an exercise dialogue.
munication awareness (users could choose to be informed via email on general activities in 2019, while in the previous venue they were only informed when their homework received feedback). Also, the exercise texts of LDM-2019 were improved or reworked by the teaching staff using comments which students left in LDM-2018. As mentioned before, homework de- livery rates could not be increased between LDM-2018 and LDM-2019, but other behaviours changed between these course venues.
Firstly, students replied more often to reviews they got from their teachers in LDM-2019 than in LDM-2018, and these “homework-feedback-dialogues” (HFDs) increased significantly in length. In both courses, HFDs were initiated by the student either with a final remark (like “thank you” or “Ok, I see”) or with an inquiry asking further questions on the exercise, the given solution, or the feedback. In both courses, “final remarks” and “inquiries” were posted approximately equally often. Notably, in LDM-2018 there were student inquiries which remained unanswered by the teachers, and this did not happen in LDM-2019.
Secondly, exercises (i.e. problem statements) received significantly fewer comments in LDM- 2019 than in LDM-2018, and these exercise dialogues (EDs) were shorter. In both courses, comments posted on exercises only contained questions regarding the exercises. These differences in communication behaviour are displayed in Table 6.1.
Thirdly, students, including those who neither delivered any homework nor left any com- ments, spent more time on the platform in LDM-2019 than in LDM-2018: In LDM-2018 an average of 6.8 active hours, and in LDM-2019 an average of 14.8 active hours was measured3.
In summary: While homework delivery rates stayed the same in both courses, the use of the platform changed. Students visited the platform more often (possibly to see homework submissions of other students and the feedback they received, as submissions made up the majority of the content available), and engaged more in feedback dialogues with their teachers. At the same time, the course’s exercises themselves seemed to have caused fewer questions than in the previous course venue, a possible result of the attempted exercise improvements.
The increase in feedback dialogues and time spent on the platform may have been caused by the improvement of communication awareness on the platform. Furthermore, it has to be noted that the learning management systems used prior to Backstage 2 / Projects did neither provide any communication functionalities to publicly discuss feedback or exercises nor the possibility to see the homework submissions of other students. Possibly, students needed time to perceive Backstage 2 / Projects as a social medium and not only as a mere homework delivery platform.
Nicol argues that teacher feedback has to be provided as a dialogue instead of a one-way communication [171]. The results above show that simply providing the students and teachers with the technology to do so (in LDM-2018) is initially not sufficient, but that a
“culture of dialogue” can emerge when simple awareness measures are implemented and students and teachers alike get used to the new functionalities.
Perspectives. The results regarding analytics-based nudging indicated that students can (in certain cases) be motivated by such nudging, but also that other motivators such as bonus points are a much more effective motivator. Arguably, further developments of analytics- based nudging should reflect how the “nudged behaviour” influences or is influenced by other factors such as the workload in other courses or external motivations.
One perspective for future developments would be to motivate students to become active early within a learning phase (for instance shortly after an assignment is given and not shortly before the deadline). Firstly, this would foster a spacing effect, which is beneficial for learning [227], and secondly, it could increase the efficiency of exercise dialogues, as it could allow more students to benefit from the answers while working on their homework. Communication on the platform happened solely between teachers and students, never did students enter dialogues with their peers. Yet, such communication could reduce the teachers’ workload [172], and future developments of the software could aim to foster such dialogues.