The first two issues with the design of the research from this study have been discussed earlier in this chapter, and are summarised in this section.
4.6.1 Generalisability
The nature of the action research utilised in this thesis may lead to potential issues concerning the generalisability of the findings from the case studies. However, it was never the intention to
extrapolate the findings of this study to a wider population or to make general claims as a result of the study. The intention was to use the findings of the action research and the case studies to inform personal professional practice and to share as an example with others in similar teaching
environments.
4.6.2 Using Summative Assessment Scores as a Measure
The problems associated with summative assessment have been discussed in chapter 3. It may seem strange to use summative assessment as a measurement in a study of the impact of formative feedback given the problems associated with summative assessment. The rationale for choosing summative assessment as a quantitative measure is based on students being motivated to engage in summative assessment, summative assessment scores being available, summative scores being a potential measure of student achievement and summative assessment scores being externally checked by external examiners.
4.6.3 Agreeing the Definition of Feedback
There are issues in trying to measure the impact of feedback when there is a range of understandings as to what feedback actually is. It has already been discussed that different students differ in their understanding of the concept feedback (Higgins et al, 2001) and there is suggestion in the literature that there is a difference in the perception of what makes good feedback between students and academics (Maclellan, 2001). Attempts have been made to take into account and discuss the possible ambiguities in the definitions of feedback, particularly in the focus group discussions.
Attempting to isolate the impact of formative feedback is complicated, especially when using summative performance and perception of feedback as measures. There are many other potential variables which could have a positive impact on student summative performance over time, such as developing learning skills, better understanding of the subject through study and practice (irrespective of any feedback provided) and increased motivation as students move along the chronological scale
of the academic calendar. There is also the possibility that students get better at assessments the more assessments they do. Therefore doing a second assignment in a module may see an improvement in the summative marks irrespective of any feedback that is provided on the first assignment or any intervening formative activities.
4.6.4 Nature of Student Cohort
In the student cohorts being studied in the first year there are also subject and programme differences to take into account which may act as variables in improving summative performance. In particular the computer forensics students had a series of potential advantages which could either individually or taken as a whole had an impact on the students’ summative performance. The programme was a new programme, certainly for the first cohort in 2005/06 and was one of only a few such courses available in the UK at the time. Computer forensics had received a large amount of publicity because of widely reported cases in the public domain, and exposure to the subject on popular TV programmes such as CSI, Cold Case and Spooks. This potentially led to a cohort of students who arguably were more highly interested in the subject and more highly motivated.
The computer forensics programme also had a custom designed laboratory which was only available to computer forensics students. The facilities in the laboratory were highly specialised and had a high technical specification. In addition although the laboratory was networked it was not connected to the university network which gave the students a feeling of independence. Specific hardware
requirements (such as write blockers – to enable read only imaging) were included in the specification of the laboratory. The laboratory was designed to enable teaching staff to emphasise the principles of evidential integrity and evidential continuity in a practical setting and to give students the opportunity to apply these principles in computer forensics analysis. In effect the laboratory acted as a base room for the computer forensics students – and the computer science students did not have a base room.
Many students studying at university have their own PCs, which are often of a higher specification than can be provided by universities. However, in computer forensics, industry standard software, such as EnCase from Guidance Software, has prohibitively expensive licences and requires a very precise technical specification to run properly. The computer forensics facility in fact addressed research undertaken by Gale (2006:1) who suggests that “in order to keep our young students at university – and these are also the most likely to drop out – we have to provide them with a work environment which is at least comparable and preferably superior to conditions they would have been
working in if they had not come to university”. It is unlikely that students would have access to similar facilities as those provided in the computer forensics laboratory, due to the complex technical specification of the laboratory and the prohibitive cost of hardware and software.
The computer forensics students also had an extra-curricular series of practitioner talks providing expert input to programmes through presentations and case studies. The regular input from practitioners, over 20 sessions were arranged in each academic year, gave the computer forensics students the impression that they were being given an extra benefit which, in turn, helped with their levels of enthusiasm and engagement with the programme.
4.6.5 Pedagogic Interest of Academics
In addition to the potential bias introduced because of the author participating in the research study, there was also a high level of pedagogic interest from academics teaching on the computer forensics programme. There are a series of pedagogical issues in the teaching of computer forensics, focussing on how to teach and assess the subject without overwhelming students with the technical, procedural, and legal aspects of the subject. The fact that students were involved in pedagogic research,
including participation in this thesis, may potentially have led to a Hawthorne type effect.
There is also a potential bias introduced by the teaching approach of the individual academics delivering the computer forensics programme. What if it is the case that the tutors who provide developmental and formative feedback are the “good” tutors anyway and they encourage and inspire students to learn from their enthusiasm, their love of the subject matter, their consideration and understanding of pedagogy and students’ learning needs? This is a particularly difficult variable to isolate.
The issues discussed above raise some interesting questions about measuring success and the comparability of measures. In order to try and tackle these issues it was decided that the comparisons that would take place in the second year of the student’s programme would only include the computer forensics students. This meant that two modules on the computer forensics programme would be analysed. The result of doing this was to negate many of the variables that arose because of the differences in the learning experience between the computer forensics and the computer science students.