film x-ray images), come to sound diagnostic conclusions, and write a clear report on the findings. Crucially, none of this involves any interaction with a patient. The Rad- DOPS, on the other hand, is a real time assessment of a doctor's performance in conducting radiological procedures, usually on fully conscious patients with whom the doctor must communicate throughout the process. I therefore decided to use the Rad- DOPS assessment for my research, as the dimension of the doctor-patient interaction that it contains seemed more intuitively comparable with the clinical work that most doctors do. This also offered the potential of comparing my findings with what had previously been found in research on other workplace-based assessments that involved doctors interacting with patients, such as the mini-CEX assessment and the DOPS assessments used in other specialties.
4.5 Study design
4.5.1 What type of study?
In setting out to answer the research question at the centre of this study, it became clear that no single method was likely to be sufficient. Consequently a mixed methods approach was adopted. However, rather than choosing mixed methods at the outset, my approach aligned with what Creswell and Plano Clark (2011) refer to as an ‘emergent’ study design (p. 54), arising out of a dynamic approach to the research rather than an approach that was pre-determined and ‘typology-based’ (ibid., p. 55). My original intention had been to focus on the content of assessors’ written feedback statements, analysing these qualitatively in order to draw conclusions about the potential educational value of the feedback provided. However the review of literature prompted me to consider other aspects of the WBA that could usefully be analysed in order to develop a more complete picture of how this formative assessment system was functioning. Features such as the timing and frequency of the assessments could add additional context and were therefore included, resulting in a more multi-faceted approach to the research. Crucially, the analysis of pairs of assessor and trainee comments was necessary in order to establish whether written feedback exchanges could be said to be dialogical.
It is important to consider the appropriateness of adopting a mixed methods approach, as it is apparent from the research methods literature that there is a range of views on whether and how methods should be mixed. On one hand, some authors take a pragmatic view of mixed methods approaches, accepting as a rationale that different methods have relative strengths and weaknesses, which may compensate for each other when used in combination. For example, Greene et al. (1989) are content to define mixed methods as any research design which uses ‘at least one quantitative method…and one qualitative method’ (p. 256). Creswell and Plano Clark (2007) expand on this definition, stating that:
As a method, it focuses on collecting, analysing and mixing both quantitative and qualitative data in a single study or series of studies. Its central premise is that the use of quantitative and qualitative approaches, in combination, provides a better understanding of research problems than either approach alone (p. 5).
Other authors are keen to emphasise a more holistic notion of ‘mixing’, thus establishing the mixed methods approach as methodology rather than method. Tashakkori and Teddlie (2003), for example, suggest that mixed methods research should be typified by mixing throughout the research process, from conception to inference, and Johnson et al. (2007) have produced an integrative definition of mixed methods research that similarly emphasises mixing at all points of the research process. This notion has not been universally welcomed, with some authors seeing the mixing of worldviews, philosophies and paradigms as being problematic. Creswell and Plano Clark (2007), for example, point out that ‘different paradigms give rise to contradictory ideas and contested arguments – features of research that are honoured but cannot be reconciled’ (p. 27). However, the mixed methods researcher, they argue, need not be derailed by these contradictions, but rather acknowledge and embrace them as ‘different ways of knowing about and valuing the social world’ (ibid., p. 27). In taking this view, which they describe as pragmatic, Creswell and Plano Clark (ibid.) contend that the research question is therefore foregrounded, with research methods and their underpinning philosophies and worldviews being duly subordinated. This was the view that I took when conducting my study, in that rather than emphasising a particular world view or paradigm and conducting my research from that perspective, I was interested in forming a view of the fitness for purpose of the formative assessment system in clinical radiology that was informed by the best use of the data at hand.
None of this is to say that a pragmatic approach permits the indiscriminate assembly of methods to qualify as mixed methods research. Rather, as Symonds and Gorard (2008) emphasise, the selection and blending of methods should be purposeful. A helpful description of five primary purposes has been provided by Greene et al. (1989), of which one – complementarity – was of key importance in my research. According to these authors, complementarity is concerned with seeking the ‘elaboration, enhancement, illustration [and] clarification of the results from one method with the results from another method’ (p. 259). This, they highlight, is different from ‘development’ (p. 259), which is a more linear use of the results from one method in order to then inform the development of another method. Rather, complementarity may take a more sophisticated form, employing some methods in parallel as well as in sequence, in a manner which is at times not easily distinguishable from the pursuit of the goal of ‘expansion’ (p. 259) – a deliberate attempt to expand the scope of the research via the introduction of additional methods. The defining feature of complementarity versus expansion or development is the rationale, which is that the range of methods used is employed in order to ‘increase the interpretability, meaningfulness and validity’ of results by drawing on the differential affordances of the different methods used. In my study, the methods utilised were selected in order to explore a number of aspects of the WBA system that could each be said to be linked to the overarching concept of ‘fitness for purpose’, thus creating a more complete, and therefore a more valid, picture of the functioning of the assessment system. The reasons for the decisions that were made about the use of each particular method, which thus gave rise to the overall research design, are discussed below.
4.5.2 Planning the research
In designing the study, I considered that I needed to establish a reference point against which to compare the assessment and feedback data that were found in the e-portfolio record in order to allow conclusions to be drawn regarding the WBA system’s fitness for purpose. Consequently, the first phase of the study involved conducting a narrative review of literature in order to establish the stated purpose (or purposes) of workplace- based assessment according to literature that included official documents and relevant academic publications. A second important function of this review was to establish what had previously been found to be effective in the implementation of formative
assessment and feedback interventions, in order to provide an objective set of criteria against which Rad-DOPS assessment and feedback data could be judged.
According to Baumeister and Leary, (1997) narrative literature reviews are useful ‘when one is attempting to link together many studies on different topics, either for purposes of reinterpretation or interconnection. As such, narrative literature reviewing is a valuable theory-building technique’ (p. 312). They contrast this with meta-analysis which, in their view, is aimed at supporting or refuting a clearly stated hypothesis, and depends on the studies which constitute it being focused on the same (or a very similar) hypothesis and exhibiting a large degree of methodological consistency.
Whilst the purpose of the literature review in my research stopped short of generating new theory, it was intended to perform the integrative function of drawing together research from different educational contexts and perspectives in order to create a coherent theoretical picture of effective formative assessment and feedback. To this end, the medical education literature was searched in order to build a picture of the current state of WBA assessment discourse within medical education. This was augmented by the inclusion of more purposively-sampled publications by notable authors in the field of formative assessment who have conducted their research primarily, although not exclusively, within the domains of school- and university-based education. This was done in order to place the concept of formative assessment in medicine within the broader formative assessment context.
The empirical aspect of the study involved a two-stranded approach: descriptive statistical analysis of a number of facets of the Rad-DOPS assessment data recorded in the e-portfolio; and qualitative content analysis of the written feedback comments of assessors and trainees. In addition to conventional statistical analysis of the coded assessor feedback statements, I chose an analytical approach described by Ragin (1987, 2000, 2008) to determine the necessary and sufficient conditions for specific types of feedback to be provided. The final stage of the study involved integrating the findings from each of the research components in order to draw conclusions and address the main research question. The activities involved in the study are displayed in Figure 4.1.
Figure 4.1 The methodological elements of the study design.
4.5.3 Which paradigm?
In order to answer the research questions, the overall approach included both qualitative analysis of features of the assessors' written feedback statements, and quantitative analysis of the resulting coding frequencies and other aspects of the assessment and feedback process. Comparisons were then made with the stated purposes of workplace-based assessment and with the characteristics of effective assessment and feedback that were yielded by the review of literature. Thus the work straddled two paradigms: the interpretive paradigm, in seeking to understand and code the written assessor feedback statements and trainee responses that comprised the raw data, and draw conclusions about the nature of the written feedback; and the
Descriptive statistical analysis of national clinical radiology assessment data ‘Ragin’ analysis of code frequencies Overall analysis and interpretation Initial review of literature Qualitative content analysis of assessors’ written feedback Classical statistical analysis of code frequencies Qualitative content analysis of trainees’ written comments
positivistic paradigm, in using descriptive and inferential statistics (such as
c
2) toexplore relationships between variables, and in using a theory-driven framework, largely constructed a priori, against which to compare the findings and to draw conclusions as to the likely effectiveness of the feedback process in Rad-DOPS assessment. These two paradigms were at times blended during the exploration and analysis processes, and in particular when using Ragin’s (1987, 2000, 2008) approach to qualitative comparative analysis (QCA). Purists may baulk at the blending of research paradigms, however some authors, such as Cohen et al. (2000), recommend a degree of paradigmatic flexibility, and caution against becoming 'paradigm-bound', in order to avoid 'stagnation and conservatism' (ibid., p. 106).