There was no known mathematical software that was able to represent or carry-
out all the actions of the three software boxes. Although there were separate software
packages that represented each of these three boxes, a mixture of software were not
• There was no guarantee that these software types would solve the same type of mathematical domain tasks, that is, finding for example three
software boxes for solving geometry or algebra
• If there were three software types representing the boxes that solved tasks within one mathematical domain, then this raised the issue whether
the students’ performance and approaches determined from the boxes
were reliable and valid since the students’ performance or approach can
be attributable to the user-design rather than ability of the software to
show and interact with steps or not.
As such, MS Excel was the software used, and it was programmed through
Visual Basic Applications (VBA) to represent the characteristics of the black-box,
glass-box and the open-box software. MS Excel was chosen as it was familiar to many
students and thus minimized the effect that familiarity with other types of software
might have on the learning of the topic. Further, using the VBA, answer sheets were
developed in MS Excel to allow students to type their answers for the tasks. This
reduced the risk of transcribing and inputting answer data incorrectly during the
analysis stage.
For the linear programming software boxes, the decision on which steps to
include in the black-box, glass-box and open-box software (see Section 4.4, p.112) and
the interface design had to be decided. The interface was dependent on the number of
Problems given, as each Problem was programmed on an individual Excel sheet. Also,
five buttons were created for each Excel sheet: ‘Input Problem’, ‘Iteration’, ‘Reset’,
Figure 7: The glass-box software for solving linear programming which was developed in Excel
The ‘Input Problem’ button allowed the students to input values. To solve the
problem, the students clicked the ‘Iteration’ button. If it was a black-box the answer
came up immediately without showing the iterations. However, for the glass-box, every
time the student clicked Iteration, an iteration was shown. The student had to click
iteration until a pop-up came up saying a solution was found. This pop-up was
necessary to ensure that the student knew the VBA linear programming procedure had
ended. The ‘Reset’ button was used for students to clear all the iterations but not their
inputted values, whilst the ‘Clear All’ button erased the iterations and inputted values.
The students used the ‘Answer Form’ button to enter their answers. There was one
3.4.4 Explanations
Whilst the student’s approach to using the software was captured via the web
cameras and screen recording in the remote-observation method and the scores from the
intervention in the experimental design, thus far, little has been said on how the self-
explanations of students were captured during the solving of the tasks. Recall from
Section 2.6 (p.32) that self-explanations are defined as explanations that students
generate for themselves whilst learning or solving tasks. The think-aloud protocol
developed by Ericsson and Simon (1984) was thus used where students were asked to
verbalise their thinking processes out loud. From this, the spontaneous explanations that
students generate for themselves were captured and analysed. Whilst written
explanations could have been used, because of the remote observation method
employed in collecting data (see Section 3.5.2, p.79), this could not be used for
technical reasons, although there was a possibility that students could have typewritten
them. However, instead when it came to students answering the problems based on the
Galbraith and Haines (2000a), students were asked to give detailed answers and this in
some way also represented the written explanations.
Further, there was concern as to whether it was necessary to record students’
actions beyond what they were doing on the computer and what they were saying, for
example recording whether they were writing on paper and reading the instructional
materials. However, based on the pilot study, recording of the students’ actions were not
necessary during the analysis if the students were able to think-aloud and say exactly
what they were doing. For students who were more reticent, the recording of their
actions allowed the researcher to know where their focus was. The recording of these
actions were taken in two ways: the researcher made written observation comments with
a time-stamp and also through the video recording of the students. The former was
reconstructed with the screen-capture videos provided further insight through
triangulation.
3.4.5 Processing Levels
In addition to the think-aloud protocol from which levels of processing could be
determined, a 10-item Approaches to Study Inventory (ASI) used on the Social and
Organisational Mediation of University Learning
(
SOMUL) project (Edmunds andRichardson, 2009; Richardson and Edmunds, 2007) was given to the students after they
solved all tasks. This instrument is referred to as the SOMUL ASI in this thesis and it is
used for quantitatively measuring whether students were using a deep or surface
processing level more predominantly during the session (Section 2.5, p.27).
The 10-items on the SOMUL ASI was chosen by Edmunds and Richardson
(2009) from an unpublished ‘Approaches to Learning and Studying’ scale developed at
the University of Edinburgh. Several inventories were considered for measuring the
deep/surface processing levels. A shorter inventory than the traditional 64-items ASI by
Ramsden and Entwistle (1981) was investigated since 64-items would be time-
consuming when added onto the experimental session which took between 1½ and two
hours.
Therefore inventories and questionnaires with a low number of items were
considered including, an 18-item ASI questionnaire (Gibbs, Habeshaw and Habeshaw,
1988), 12-item SOMUL ASI questionnaire (Richardson and Edmunds, 2007), 20-item
Study Process Questionnaire (SPQ) by Biggs, Kember and Leung (2001) and the 26-
item Approaches to Learning Mathematics Questionnaire (ALMQ) (Crawford et al.,
1998a; 1998b). The shorter questionnaires were more satisfactory as they were less
time-consuming and hence the 20-item SPQ and 26-item ALMQ were eliminated. The
previously questioned by Richardson (2000). Although, there were concerns on whether
the 12-item SOMUL ASI questionnaire would be sufficient for determining whether
there was deep or surface processing levels, Richardson and Edmunds (2007) showed
that the items loaded satisfactorily on the factors of deep and surface processing levels,
except for two items each relating to either a surface or deep learning. These two items
were dropped and a 10-item SOMUL ASI was used for this research instead.
3.4.6 Explorations
Whenever students used the software boxes for testing numbers or decided to
redo a procedure to examine a process, this was termed ‘exploration’ (Section 2.7,
p.38). Only for the mechanical tasks were students required to use a software box. The
constructive tasks were devised to be solved by either pen-and-paper or a software box.
The interpretive task was devised to be answered solely by pen-and-paper.
By examining the students’ videos and observing what they were doing for each
task, students were either coded as not-exploring (0) or exploring (1) for each task. Each
video was checked at 5-10 second intervals to determine how the students were using
the software. For mechanical tasks, students were coded as exploring if they used the
software boxes for any other purpose besides solving the given mechanical task such as
solving using a different number. In the case of the open-box software when students
tested a different sequence of processes whilst solving the tasks such as using a different
pivot variable, this was also coded as exploring. If the students used the software boxes
for either the interpretive or the constructive task this was coded also as exploring.
3.4.7 Mathematics Confidence
As noted in Section 2.8 (p.42), mathematics self-confidence is used as an
attitudinal measure in this thesis. Bandura (1986) pointed out that self-confidence
as students’ evaluation of their prior experiences influences how they will perform in
future tasks.
There are several instruments for measuring mathematics confidence including
the Mathematics-Computing Attitude Scales (MCAS) by Galbraith and Haines (2000b)
or the Mathematics Confidence Scale (MCS) by Fennema and Sherman (1976). These
instruments have a high level of internal consistency but use at least 30 items. However,
during the 2-hour session, the experimental intervention required the students to learn
linear programming, understand a software box and then solve nine tasks in linear
programming using the software box. A scale was thus needed that will ensure the least
fatigue for measuring mathematics confidence before the students proceeded to the
experimental intervention.
Bandura (1986) has suggested that students are quite capable of judging their
own levels of confidence accurately but he cautioned that the assessment of confidence
should be tied closely to the mathematical topic rather than a global assessment as in
this case general mathematics. The linear programming topic was chosen because of its
unfamiliarity to students. Whilst algebra was considered to be the closest topic to linear
programming, some tasks required the use of logic and understanding word problems.
Therefore, a general assessment of mathematics confidence was used to encompass all
these areas.
Students were asked to assessed their level of mathematics confidence on a scale
of 1 to 10, where 1 = low confidence and 10 = high confidence. Collins in Bandura
(1986) also used a similar method where he asked students to assess themselves on
whether they had high and low mathematics confidence and he found through using this
method that students’ performance was related to their assessed confidence levels. As
this simplified scale was used, students were also asked to assess their computer
3.4.8 Performance
Performance was measured by the marks acquired by students for each task. A
marking scheme was developed for all problems and tasks and is presented in Appendix
6 (p.313). For the mechanical tasks, all students were required to solve these via the
software boxes. As the researcher ensured that all students inputted the correct values
for the mechanical task, this meant all students got these tasks correct. It was imperative
that students solved the mechanical tasks correctly to ensure they stood a chance of
solving the interpretive and constructive tasks since both the interpretive and
constructive tasks were linked to the mechanical task solution. Therefore, scores from
mechanical tasks were not included when investigated performance differences between
the software boxes.
3.5 Remote Observation
The remote observation method was used for collecting data in this study. This
section deals with why remote observation was chosen and a brief overview of the
remote observation process. This process is also further discussed in Chapter 4.