MRCHIPS is an agent-based solution to the problem of mentoring novice- programming students. The MRCHIPS architecture allows for the reactive and deliberative reasoning required for the agent to operate within a dynamic desktop environment while making diagnostic decisions about programming errors. The Beliefs Desires and Intentions (BDI) based planning system is used to coordinate operations within the agent from responding to inputs, controlling outputs and scheduling the other reasoning resources in the agent. Reactive reasoning is supported in the BDI by maintaining an agenda of goals and selecting an appropriate plan to solve the goal. Deliberative reasoning in the BDI is supported using a
174
series of plans refine the steps of a goal before ultimate solution is found. Most of the agent’s deliberative reasoning involved diagnosis of novice errors. Categorising errors into different types lead to their consideration as individual diagnostic situations, which provided a strong correlation to their representation as cases and indicated the use of case-based reasoning (CBR) in the agent. However not all of the deliberative reasoning is processed by the CBR; the diagnosis of syntax errors are processed using a rule-based parser. The syntax for programming languages are usually defined as a series of rules, such as in the BNF notation used in chapter eight in table 8.1. The rule representation therefore naturally lends itself to efficient processing in a rule-based parser. The agent-based solution allows different reasoning methods to be applied to perform different types of problem solving. The MRCHIPS agent met its initial design requirements. It was able to monitor the learner’s activity, accurately diagnose the errors and respond to the learner in a timely manner. This was possible even though the architecture is run from an interpreter, PAL, within another interpreter, Python. Much of the speed and accuracy in reasoning is gained via use of the lookup tables, built from the Python hash table data structure, which allow for the fast indexing of data and reduces the need to search. The BDI uses a hash table to reference to the underlying Python functions that implement the PAL interpreter. The CBR also makes use of hash tables to form the discrimination network so indexing is performed via a single lookup for each argument of the case. Operations that are time consuming such as unification and the BNF parser, both of which involve a systematic search through data structures, are used sparingly.
However there are a few limitations in the operation of MRCHIPS, features that it is not able to carry out or that would require some redesign to implement. The limitations examined detail below and are related to:
a) Strategic – how the agent informed the learner, only monitored the learner in some modes and no direct modelling of the learner; b) Technical – no capability to adapt to unforeseen situations;
175
c) Social – the limited capacity for natural language interaction; d) Portability – only available on Windows platforms.
Two flaws discovered during evaluation had to be corrected to before further experimentation could take place. First was the strategy used by MRCHIPS to inform the user that it had a solution to an error. On detecting an error MRCHIPS waits for a period to allow the learner to self correct if possible. In its original configuration MRCHIPS provided no feedback that it had detected the error and appeared only after the delay. Feedback from the learners indicated that this was disconcerting so a semi-transparent pop-up window was introduced to alert the learner that MRCHIPS would provide help after a delay. The second was that the agent only monitored the environment when the MRCHIPS character was minimised. As some learners preferred to work while MRCHIPS was on the desktop they were unable to receive further assistance. It was incorrectly thought that whenever the agent character was on the desktop the learner would be in dialog with it, so they were unable to proceed with writing code. Fortunately the solution required only changes in the plans within the knowledge base to allow scanning of the environment to be performed as a separate intention, effectively running as a thread and irrespective of the state of the agent character.
The MRCHIPS strategy only allows for indirect modelling of the user by modelling the types of programming errors. No attempt is made to directly model the user in the way that a system such as the Genie intelligent assistant, reviewed in chapter 3, is able to do. Modelling the user would involve making an assessment of the user’s level of expertise and adjusting the behaviour of MRCHIPS to suit the user’s preferences. For example a novice user might prefer help only in the form of the solution to an error, but once more accomplished he or she might prefer a longer explanation to the cause of the error. Modelling the user via programming errors was adequate for experimentation with MRCHIPS but for longer term use direct modelling of the user would allow the agent to track the user’s progress, present information in a format that is
176
tuned to the user’s ability and allow for the more complex social interactions.
The CBR in MRCHIPS has no capacity to automatically acquire new cases this would have provided MRCHIPS with a form of learning and the capability to adapt to novel or unforeseen situations. There are two areas of the agent architecture able to support learning but they were not required for the evaluation. First the function of the CBR could be extended to implement the adaptation and storage operations for new cases. The BDI plans could be used to guide the adaptation process, which would require the manipulation of the Prolog data structure used to represent the case. The second learning capability is an autobiographical memory, which would allow the agent to consult the record of its experiences for decision-making and reflection. The journaling system already records the decisions of MRCHIPS but the agent makes no further use of the information. Autobiographical memory would allow MRCHIPS to model the record of individual learners and adjust decisions to meet their needs.
MRCHIPS has limited capacity for complex social interactions with the user, which could be used with the diagnosis of logical errors and to offer messages of support and encouragement. No method could be determined to allow MRCHIPS to diagnose the cause of logical errors because the program code would be legal and so the agent would need to understand the programmer’s intentions for the code. A solution was designed to have the agent guide the learner through a question and answer process and offer suggestions to allow them to determine the cause but this was not implemented. There are a few plans and cases in the agent’s knowledge base that offer messages of encouragement, but these are presented at random. The use of autobiographic memory would allow messages to be tracked and encouragement could then be offered within a strategy.
MRCHIPS is only currently able to run on Microsoft Windows based platforms. The reasoning subsystems the BDI planner, CBR and BNF
177
parser are platform independent but the agent interface subsystem is specific to the operation of the WIN32 programming interface and the animated character relies on the Microsoft Agents engine, which is only available for Windows. Converting the agent interface to work with other GUI systems should be possible if the appropriate operating systems resources, such as system events and messages, are accessible. An alternative to the Microsoft Agent character interface would also be required such as Double Agent or that used in the Adele system reviewed in chapter four.
10.3 Summary
A quantitative evaluation for the effect of MRCHIPS on the work of novice programmers has been given. The data presented in this chapter provides evidence for the effective support of a pedagogical agent for assisting novice programming students as they learn Python as a first language. The results of the experimentation were able to demonstrate that the presence of the agent was able to assist participants to make progress with developing a Python program, not least because MRCHIPS was able to provide answers. Comparing the results from the groups of novice programmers, those working with MRCHIPS were 10% more productive than those working with no agent. From the t-test the calculated value t = -2.505 was found to exceed the critical value of t = + 2.110 at p = 0.05 with a df = 17. Therefore the null hypothesis is rejected and it is concluded the mean score for the mentored novice students (65.5%) was significantly higher than for the un-mentored novice students (55.5%). Analysis of the experimental findings show there to be a significant correlation between the presence of MRCHIPS and the improvement in performance of the novice programmers. There was a positive correlation coefficient of 0.73 between the support offered by MRCHIPS and the grade achieved by the mentored students. There were also indications of learning where subjects were able to recognise and solve problems without the guidance of the agent, although the correlation coefficient of 0.21 was less significant. The evaluation was also able to show some support for learning in that four of the mentored students were able to recognise and solve one or more of the repeated
178
errors without the aid of the agent. The mentored student who was able to solve the logical error even though MRCHIPS had no support suggested he recalled some knowledge from an earlier programming experience and replied, “It just seemed to be the way it worked.” The major caveat with the results is that the size of the study group was small and the study was short in duration. Therefore the effect of an individual’s performance on the reading would have a disproportionate effect on the findings. It had originally been planned to then run a larger study over a longer learning period. Unfortunately due to a change in employment that required a move away from the university contact with the student study group was lost.
179