Librer´ıa de planificaci´ on Optaplanner

Complex classification, such as modelling non-verbal behaviour, has been demonstrated using machine learning (Buckingham et al., 2012, 2014; Graves et al., 2007; Hai et al., 2015; Kahou et al., 2016; Khandait et al., 2011; Lopes et al., 2017; Mayya et al., 2016; Owayjan et al., 2016; Pinheiro and Collobert, 2014; Rothwell et al., 2006, 2007; Teng and Yang, 2016; Tompson et al., 2014).

6.5 Classifying cognitive states from non-verbal behaviour 115

Such algorithms learn the problem domain using statistical analysis of the data,

producing a model with hyper-parameters optimised to describe the classes, or patterns, within the source data. The trained model can then use these learned parameters to recognise familiar patterns in new data.

Breakthrough research (Rothwell et al., 2006, 2007) from the field of de- ception detection has demonstrated the effectiveness of ANN (Chapter 5) for classifying deception from non-verbal behaviour during interrogative interviews. In recent research (Buckingham et al., 2012, 2014) the approach has been successfully modified to classify comprehension in both face-to-face medical interview and student oral examination contexts. Importantly, the approach de- pends entirely on the subject’s expressed non-verbal behaviour during cognition. The generalised model of learner comprehension-indicative NVB demonstrated in Buckingham et al. (2014) requires no pre-planning of visual context or question content.

In work by both Rothwell et al. (2006, 2007) and Buckingham et al. (2012, 2014) participants were recorded while answering questions orally, under in- terview conditions. The video recordings were analysed computationally to extract a numerical model of learner non-verbal behaviour for a given time frame. The model was then classified using an MLP (Chapter 5) to detect the presence of a target cognitive process. The research indicates the MLP classifiers performed well, with Rothwell et al. (2006) reporting classification accuracy of 74% on deception and Buckingham et al. (Buckingham et al., 2014) reporting 76% classification accuracy on comprehension.

The success of the approach pioneered by Rothwell et al. (2006, 2007) and adopted by Buckingham et al. (2012, 2014) was partly owed to the high fidelity of the behavioural data model. In Buckingham et al. (2014) a classification was made on every 1 second of video. Each video frame in the 1 second tranche was analysed to produce a vector of 40 binomial ±1.0 variables. For example,

116 Modelling and classifying patterns of non-verbal behaviour

+1.0 if the left eye was fully open, -1.0 if it was not. The matrix of values was then reduced to a single 40 variable vector using summary statistics.

6.6

Conclusion

The review of literature presented in this chapter has highlighted how machine learning, particularly artificial neural networks, can be used to model patterns of behaviour expressed during learning activities.

Literature has shown that non-verbal behaviour, such as stress responses, is not dependent on social interaction with a human. This finding highlights an opportunity for techniques, found in literature on analysis of human-to- human communicative behaviour (Buckingham et al., 2014; Khandait et al., 2011; Rothwell et al., 2007), to be applied in the classroom to analyse learner behaviour during e-learning.

Literature has provided a feature set of behavioural features, shown to be linked to cognitive processes, including head movement, posture, gaze and skin tone change. Literature has also shown that streams of image data can be analysed using Haar cascades and artificial neural networks to produce a numeric vector representative of learner behaviour over discrete windows of time.

The literature presented in this chapter demonstrates that learner non-verbal behaviour, such as that used by human tutors to assess comprehension in a classroom setting, can be extracted from digital information and automatically classified to inform on the cognitive state of a learner during a learning activity. The literature suggests an opportunity to develop a similar technology capable of classifying learner comprehension in near real-time, during learning, as a method of timing micro-adaptations in conversational tutoring.

Chapter 7

Hendrix 1.0: A conversational

intelligent tutoring system for

programming

7.1

Introduction

Hendrix 1.0 (Holmes et al., 2015a) is a conversational intelligent tutoring system (CITS) designed to tutor computer science and programming. At the heart of the CITS is a conversational algorithm which has to ability to interpret and respond to discourse using natural language. Using natural language interpretation, Hendrix 1.0 mimics a human tutor by guiding the learner through tutorial content using discourse, challenge and feedback. It converses with a learner to identify gaps in knowledge through questioning and expanding the curriculum when gaps in knowledge are identified. Hendrix 1.0 supports learners by detecting questions and providing definitions and examples. Hendrix 1.0 uses both syntactic and semantic language analysis to extract and match information from learner utterances. Its two loop algorithm is dependent on identifying the short term goal of a learner in each conversational turn.

118 Hendrix 1.0: A conversational intelligent tutoring system for programming

Hendrix 1.0 makes both technical and educational contribution to the field of intelligent adaptive CITS. The key contributions of the system are discussed in section 7.3.1.

This chapter introduces Hendrix 1.0 (section 7.3), details the requirements and challenges in developing the system (sections 7.3.2 and 7.3.3), provides wire frames and explanation of the user interface (section 7.4.1), presents the architecture of the system (section 7.4) and discusses the core functions of the system (sections 7.4.2, 7.4.4 and 7.4.3).