Though not initially planned, the BRACElet5 organisers soon began to understand the project’s operation in terms of action research. Action research is frequently implemented as a cycle of plan, act, observe and reflect (De Villiers, 2005; Zuber-Skerrit, 1992). The first action research cycle built
theory about novices’ acquisition of programming skills in terms of the taxonomies of Bloom and the Structure of the Observed Learning Outcome (SOLO) (Biggs & Kollis, 1982), while the second action research cycle investigated the relationship between code reading and code writing (Clear, Edwards, Lister, Simon, Thompson & Whalley, 2008). The third action research cycle continued investigating statistical relationships between reading and writing code, as well as whether programming consists of a hierarchy of knowledge and skills (Whalley & Lister, 2009).
The participants in the first BRACElet workshop, held in December 2004 in New Zealand, considered the work of the Leeds Group to lack a base of learning theories or educational models. Consequently, on reflection, they devised an instrument in the form of a set of multiple choice and short answer questions (Whalley & Clear, 2007; Whalley & Lister, 2009), based on the revised Bloom’s taxonomy (Anderson, Krathwohl, Airasian, Cruikshank, Mayer, Pintrich, Raths & Wittrock, 2001). These questions were included in the exams for the following semester at some of the participating New- Zealand institutions, and the data the data obtained were analysed at the second BRACElet workshop (Clear, Edwards et al., 2008; Clear et al., 2009; Whalley & Clear, 2007; Whalley & Lister, 2009; Whalley & Robbins, 2007).
During the second BRACElet workshop at the 18th NACCQ conference of New-Zealand in 2005, the SOLO taxonomy (Biggs & Kollis, 1982) was introduced and consolidated as part of a toolkit for research at the partaking institutions. This was an attempt to gain a better understanding of the students’ responses to the levels of difficulty of the multiple choice and short-answer questions (Clear, Edwards et al., 2008; Clear et al., 2009; Whalley & Lister, 2009; Whalley & Robbins, 2007). The results were published in several papers (Lister et al., 2006; Philpott et al., 2007; Thompson, Luxton- Reilly, Whalley, Hu & Robbins, 2008; Thompson, Whalley, Lister & Simon, 2006; Whalley, 2006; Whalley, Lister, Thompson, Clear, Robbins, Kumar & Prasad, 2006). See the next section on BRACElet’s findings for a discussion of the results.
At the third BRACElet workshop in March 2006, at Auckland University of Technology (AUT) in Auckland, Australia, a common ‘prototype’ framework was developed to allow participants flexibility in their research as well as to compare and contrast experiments. Three essential components were included (Clear, Ewards et al., 2008; Clear et al., 2009; Whalley & Clear, 2007; Whalley & Robbins, 2007), namely –
• a reading and tracing component to measure students’ ability to read and understand code, using Bloom’s taxonomy to judge the difficulty;
• a SOLO component to classify a student’s ability to abstract, typically with a question to ‘explain in plain English’; and
• a writing component where students write some code comparable in difficulty to one of the reading questions.
During the fourth workshop at the 19th annual NACCQ conference in Welllington, New-Zealand, in July 2006, the common framework developed at the third workshop was reviewed and extended to further investigate the writing skills of novice programmers. The framework now included four components (Clear et al., 2009; Whalley & Robbins, 2007) namely –
• Bloom’s taxonomy to categorise the cognitive process required to answer a question; • the SOLO taxonomy to classify the reasoning level expressed in answers;
• a categorization of the question-types used in BRACElet; and
• analysis of student script annotations to provide insight into the processes students follow to solve a problem.
Both the fifth and sixth workshops were used to reflect and plan for the second action research cycle. During the fifth workshop at the 20th annual NACCQ conference in Nelson, New-Zealand, in July 2007, questions fitting the common BRACElet framework were set (Clear et al., 2009). A new type of question, Parsons’ problems (Parsons & Haden, 2006), was included. Parsons’ problems consist of jumbled code which should be placed in the correct order. The sixth workshop at AUT, Auckland, Australia, in December 2007, was sponsored by the ACM and SIGCSE and attended by nine institutions. Contributed data were analysed to develop new research instruments to allow the evaluation of novices’ program-writing skills in order to investigate a correlation with their program- reading skills (Clear, Edwards et al., 2008; Clear et al., 2009).
The seventh workshop in January 2008, at the ACE2008 conference in Wollongong, Australia, initiated the action for the second BRACElet action research cycle. The goals for this action research cycle were to consolidate ‘explain in plain English’ questions; generate new questions; relate answers to SOLO levels; examine gender effects, as well as differences between international and local students, and undergraduate and postgraduate students; and to develop an ideal examination paper (Clear, Edwards et al., 2008; Clear et al., 2009).
At the eighth workshop in July 2008, at the NACCQ conference at AUT, Auckland, Australia, analysis of the assessment data from novice programmers contributed by participating institutions, resulted in further refinement of the reading and writing classifications in the SOLO taxonomy (Clear et al., 2009).
The ninth BRACElet workshop at ICER in Sydney, in September 2008, was used to reflect on the data analysis up to that point, and for preliminary discussions on the third action research cycle (Clear et al., 2009; Whalley & Lister, 2009). BRACElet regarded code writing ability as the dependent variable in their investigation into a hierarchy of knowledge and skills in programming, as code writing is considered to be at the top of the hierarchy. The third action research cycle continued
searching for statistical relationships between code writing and the other skills considered to be pre- cursors to code writing (Whalley & Lister, 2009).
At the tenth BRACElet workshop in Wellington, New-Zealand, during the ACE2009 conference in January 2009, the BRACElet 2009.1 (Wellington) Specification was issued. This Specification published the research plan for the third BRACElet action research cycle and invited new participants to join (Clear et al., 2009; Whalley & Lister, 2009). The common core of the BRACElet 2009.1 (Wellington) Specification consists of three parts, Basic Knowledge and Skills; Reading/Understanding; and Writing. The Basic Knowledge and Skills part should test all the programming constructs used in the other two parts to establish whether students understand the programming constructs and have mastered relatively concrete skills such as tracing. The Reading/Understanding part should determine whether students can recognize the purpose of a piece of code from reading the code (that is, are they able to abstract?). At least one of the following types of questions should be used here: ‘Explain in plain English’; Parson’s problems or Code Classification Questions (placing pieces of code into two groups so that the code pieces in a group are similar in some way, and different from the code pieces in the other group). For the Writing part, at least one code writing task should be of similar complexity to one of the tracing tasks, and likewise, at least one code writing task should be of comparable complexity as a Reading/Understanding part (Clear et al., 2009; Whalley & Lister, 2009).
The eleventh BRACElet workshop was held at ITiCSE in Paris in July 2009. The group that attended this workshop, replicated earlier BRACElet studies at other institutions to include a far larger pool of naturally occurring data; refined the SOLO-taxonomy for code-explaining questions; extended the SOLO taxonomy to cover code-writing questions; expanded earlier studies on student annotations (‘doodles’) while answering examination questions; and explored a theoretical base in mathematics education for the graduated hierarchy of skills development in programming. This confirmed earlier empirical evidence about the relationship between code explaining and code writing, although the exact nature of the relationship remains unclear (Clear et al., 2011; Lister et al., 2010; Lister & Edwards, 2010).
The twelfth BRACElet workshop was held in Brisbane during January 2010 at the ACE2010 conference. A number of BRACElet activities (or ‘threads’) for the following year were presented (Clear et al., 2011; Lister & Edwards, 2010). Two of these are incorporated in the BABELnot project (Lister et al., 2012) that continues BRACElet’s work, namely –
• a survey and analysis of the types of exam questions used in examination papers for novice programmers, including interviews with academics on how it is done; and
• an analysis of longitudinal data for students from the introductory course to the final capstone project.
The final BRACElet workshop was held in September 2010, at the AUT, with a two-fold purpose, namely to extend the work on theory that began in Paris in 2009; and to launch an on-going research programme into the development from novice to professional programmer (the ‘NExpertise’ project). The workshop divided into three groups that focused on the following issues (Clear et al., 2011):
• Investigating a hierarchy in the development of novices’ understanding of the relationship between the concepts ‘object’ and ‘class’.
• An investigation into novices’ misconceptions about assignment.
• A longitudinal study on individual students to determine whether exam results in introductory programming courses can predict success.
As can be seen from the above and mentioned before (see section 3.2) BRACElet conducted an extensive multi-institutional, multi-national study into novices’ acquisition of programming skills over three action-research cycles. The first four BRACElet workshops constituted the first action research cycle, while the fifth to the eighth workshops covered the second action research cycle (Clear, Edwards et al., 2008), and the ninth to the thirteenth workshops the third cycle (Clear et al., 2011). The most important BRACElet findings are discussed in the next section.