A summary of the findings and contributions in relation to the research aim of exploring the feasibility of building an ATS using unobtrusive sensors for the learning of computing programming is listed below.
1. To develop a method for sensing the emotions of students that are detrimental to learning on a moment-to-moment basis from unobtrusive and cost effective sensors.
155
The use of unobtrusive sensors (keyboards, mouse and web cameras) in affect detection for the context of a tutoring system that tutors students in computer programming has been established.
2. To investigate the use of multimodal affect sensing techniques for enhanced affect detection.
A multimodal system of affect detection using keystrokes, mouse clicks, contextual logs, facial and head postures combined using various fusion techniques was proposed and investigated.
3. To develop a method for formulating and adapting the tutoring response in accordance with the sensed emotions of the students for optimizing their learning. A pedagogical based model of responding to the students expressing negative affect centered on a context sensitive and comprehensive hint system was proposed and implemented.
4. To design an architecture and implement an ATS for the learning of computer programming.
An architecture was proposed and the ATS was implemented and its effectiveness and acceptance evaluated through an experimental study and a series of focus group discussions.
In this research, I have explored the use of unobtrusive sensors in affect detection for the context of a tutoring system that tutors students in computer programming. More specifically, I have established the viability of using keystrokes, mouse clicks and contextual logs for the detection of frustration on a level of granularity that is adequate for timely remedial intervention. Keystrokes and mouse clicks were traditionally used in computer security domain for authentication and user identification purposes and were rarely used for affect detection and hence, the significance of the use of keystrokes and mouse clicks for affect detection in this research.
Another area of significance is in multimodal affect detection – the fusing of multiple sensing channels’ outputs. It is a fact that human emotion is expressed in various channels e.g. facial, vocal and bodily expressions but implementation of a multimodal affect detection
156
system is still rare in occurrence (Jaimes & Sebe, 2007). In this research, a multimodal system of affect detection using keystrokes, mouse clicks, contextual logs, facial and head postures combined using various fusion techniques was proposed and investigated. It was further verified that a multimodal fusion of the proposed sensors (AUC=0.914) outperformed the best unimodal channel (the facial channel with an AUC of 0.85). Although the features that contributed most to the accuracy of the multimodal model were the ones derived from head postures and facial channels, the keystrokes and mouse clicks filled in for the periods of no detection when both the head postures and facial features were not available. To illustrate, facial features were missing for 17% of the instances across all students for my study which is a semi-naturalistic setup where students were urged to remain in their seats for the web camera to get a full frontal view of their face for the entire session. In a naturalistic environment with lesser constraints imposed on the students, the availability of the facial features would likely be further degraded and thus the need for the keystrokes and mouse clicks to complement the lack of detection of facial features.
The next area of significance relates to the implementation of an Affective Tutoring System which tutors students in the domain of computer programming. Despite the recognized benefits of incorporating affect into tutoring systems, Affective Tutoring Systems are seldom implemented (Thompson & McGill, 2012), let alone one that tutors in computer programming. In this research, a pedagogical based model of responding to the students expressing negative affect centered on a context sensitive and comprehensive hint system was proposed and implemented. In addition, the architecture of the tutoring system also caters for future extension into other tutoring domains, with loose coupling between the various subsystems and pluggable modules within each subsystem.
Lastly, the effectiveness of the Affective Programming Tutoring System was also evaluated through an experimental study and a series of focus group discussions. The results of the experimental study showed that the students who were tutored with the full affective version of the APTS were able to attempt more exercises and were able to complete the exercises within a shorter period of time as compared to those tutored with the non-affective version, thus supporting the hypothesis that the full affective version of the Affective
157
Programming Tutoring System resulted in more effective tutoring as compared to the non- affective version. Analysis of the transcripts of the focus group discussion revealed that the students in general rated the usability and their acceptance of the tutoring system favorably. In particular, the filling in of missing lines of codes feature and the hint function of the system were highlighted as the top two useful functions of the system. The students’ general consensus was that the filling in of missing lines of codes feature substantially reduces their cognitive load in the computer programming learning process and that the hints function offers them context sensitive help to overcome the programming hurdle.