3.3.1 Sampling
My sample is drawn from the population of teachers in primary schools in England and the sampling techniques were driven by the RQs. This study is based on two kinds of sample: one set of submissions from a pre-existing database and two schools where practice was examined over a period of two or three years. This section will outline each sample, before moving on to a more in-depth discussion of the case studies, which form the larger
proportion of the thesis.
In order to answer RQ1, regarding practice in science assessment, I utilised a pre-existing dataset from the Primary Science Quality Mark (PSQM). PSQM requires the science subject leader (SL) in each school to reflect upon and develop practice over the course of one year, then upload a set of reflections and supporting evidence to the database to support their application. One of the 13 PSQM criteria (C2) requires the subject leader to explain how science is assessed within the school, so it was analysis of the evidence submitted under criterion C2 that formed the basis of this initial analysis in DBR Phase 1 since this could provide a ‘snapshot’ of approaches taken by English primary schools to the formative and summative assessment of pupils’ learning in science.
The PSQM dataset allowed a sample of schools from across England to be utilised to answer RQ1, however, it could not be considered a representative sample since the schools were self-selecting, which could mean that their practices would be different to other schools. They were working towards the Primary Science Quality Mark which required them to reflect upon, and perhaps develop, their assessment practices, so it is likely that non- sampled schools may have had less developed assessment practices. In addition, the
65 reported practice may have been presented in a positive light, in support of their award application. Nevertheless, sampling can be described as a balance between what is ideal and what is possible (Newby 2010) and whilst it is acknowledged that this is only a subset of ‘interested’ schools, the PSQM dataset was closely matched to RQ1, providing a framing of the issues for this study and for the beginning of the TAPS project, the first Phase of DBR. It also led to a realisation that RQ2 would need a different approach: “a need exists because one data source may be insufficient” (Cresswell and Plano Clark 2011: 8). The relationship between formative and summative assessment was not explicitly described in the C2 reflections; exploration of this relationship required interaction with participants.
In order to study the conceptualisation and enactment of ‘formative to summative’ assessment (RQ2) and its development over time (RQ3), there needed to be in-depth and ongoing analysis of practice in action. The use of case study within DBR will be discussed below, the focus here will be on the sampling of the schools. Two schools were selected as type exemplars, selected to illustrate what is possible (Newby 2010), a purposive or critical case sampling of schools (Teddlie and Yu 2007). The cases were not selected to be
representative, but to be informative (Cohen et al. 2011), this purposive sampling was driven by the research questions which required exploration of change over time; the goal was depth rather than breadth of information-rich cases (Mears 2012). These are special cases, TAPS project schools for which I was the link tutor, enabling me to gather data about their assessment practices over time. Participation in an in-depth study over two or three years already suggests that these schools are atypical; such participation requires the support of the head teacher and the SL for repeated school visits and project days. Such commitment to remaining an active member of the project is likely to depend on science being given high priority in the school, so in effect I have only looked at schools where science is ‘strong’, but to answer my RQs I need the ‘right source’ (Newby 2010) which could commit to long term involvement.
3.3.2 Case study within DBR
In order to understand what is happening during DBR research when teachers try to use formative assessment information to make summative judgements, there needs to be a
66 detailed study of the process. Sufficient data is required to explore potential significant features of the case, there needs to be ‘thick description’ (Geertz 1973). Case study is in- depth study of a situation in its natural setting which is ‘strong in reality’ (Adelman et al. 1976: 148). For me this does not mean a single positivist objective reality waiting to be discovered, it sits within an interpretivist constructed reality where different people have different perceptions (Bassey 1999), whether that be the teachers, the researcher or the readers of research. A case study includes: "the study of an instance in action" (Adelman et al. 1976: 141) and it is understanding of the ‘in action’ element which is so central to DBR and to this study of assessment.
A case study is the: “study of singularity conducted in depth in natural settings” (Bassey 1999: 47), with a ‘singularity’ being the particular event, practice or situation under scrutiny. Adelman et al. (1976) call the case a ‘bounded system’ which is selected as an ‘instance of a class’, for example, a particular school as an instance of primary schools in England. There are two ways of setting up a case study, either a hypothesis leads to a case being selected, or a case leads to a hypothesis, the case study is a ‘step towards theory’ (Stake 2006). This research is an example of the former: the hypothesis that formative assessment could support summative assessment led to the identification of schools which were working towards this. Of course, behind the hypothesis were previous experiences with schools which helped develop the hypothesis, but this pre-dates this study. In this research the cases were chosen to support development of the understanding of the ‘class’ of assessment in primary science.
The iterative cycles of DBR can be analysed using case studies, with the role of the
researcher acknowledged as a potential variable within the enquiry: by asking questions and observing they may change the situation they are studying (Bassey 1999: 43). It could be argued that DBR goes further than noting the influence of the researcher, it is an explicit aim of DBR that the situation be developed in collaboration with the teachers. Case study can provide an in-depth consideration of the process: case studies are: “a step to action. They begin in a world of action and contribute to it” (Adelman et al. 1976: 148). Walker (1983: 163) notes that case study research may be accused of trying to ‘embalm practices’ which are always changing. Perhaps the iterative cycles of DBR can begin to address this by placing
67 a pause for reflection during each cycle. The changing practices of teachers within the study provide another layer for analysis, reinforcing the need for the depth of data collection which is an essential feature of a case study.
3.3.3 Case studies in this research
Stake (2006) suggests multiple case study research seeks to understand the ‘quintain’ (equivalent to ‘class’ above) by observing in multiple situations. The cases are studied for what they reveal about the quintain, however, Stake is keen to stress that multi-case
research is still primarily concerned with the case; it is not a simple comparative study which looks for similarities and differences on a small number of attributes (Stake 2006: 82). I feel that my research does not meet Stake’s multi-case criteria, for although I am keen to
understand the detail of what is happening when teachers assess, I am using case study as a method for testing and developing theory, rather than aiming to develop a better
understanding of assessment in that particular school. Thus although multiple case studies are utilised in this research, each case is used for a different purpose, as shall be explored in the next section.
The case studies contained in this research focus on different periods within the DBR process and are presented for different purposes, perhaps making them different types of case study. Stake (2006) splits case studies into ‘intrinsic’, where the researcher is
interested in the case for its own sake, and ‘instrumental’, where interest in the case is driven by an outside concern. Both of my case studies can be placed into Stake’s ‘instrumental’ case study category since they have been selected as ‘test-beds’ for the model of ‘formative to summative’ assessment proposed as part of the TAPS project. Yin (2014: 238) divides case studies into: ‘exploratory’, whose purpose is to identify research questions; ‘descriptive’, which describe the phenomenon in its real world context; and ‘explanatory’ whose purpose is to explain how or why a condition came to be. The last category appears to be the closest to my case studies as I am trying to explore how formative assessment information can be used for summative purposes.
68 Bassey (1999: 62) identifies ‘theory seeking’ and ‘theory testing’ case studies and it is on this categorisation which I can separate the two case studies in this research. Case A is ‘theory seeking’ since it is focused on the Development DBR phase, at the stage when theory is being developed. Case B is ‘theory testing’, since it is focused on the
Implementation DBR phase where the TAPS pyramid model is being used in school. The iterative cycles of DBR call into question the separation of the cases in this way, since with each new piece of information theory seeking and testing could be taking place, however, by categorising the cases in this way I am providing a shorthand for their purpose rather than a full description of all elements.
DBR methodology requires that such case studies result in useful products for researchers and practitioners, which appears to be at odds with the typical aim of a case study where in- depth understanding of the case is the aim. If it is not possible to have freedom from time and context then it could be questioned whether it is possible to make generalisations; added to this: “the trouble with generalisations is that they don’t apply to particulars” (Lincoln and Guba 185: 110). Simons (1996) describes this as a paradox, focusing on the unique whilst seeking to generalise. She suggests that the research should challenge the reader to construct the generalisation for themselves: “construct their own meanings from the evidence we offer” (Simons 1996: 232). Rather than seeking statistically generalisable positivist ‘truths’, the reader applies the assertions to their own situations, making
‘naturalistic generalisations’: “the responsibility for making generalisations should be more the reader’s than the writer’s” (Stake 2006: 90).
The concern for passing the ‘responsibility’ to the reader is that the teachers and policy makers who are able to act on the lessons learned from a case study will not have the time or means to access the full reports. Bassey (1999) suggests ‘fuzzy generalisations’ can act as sound bites from research: “Do y instead of x and your pupils may learn more” (p51). Such summary statements provide accessible conclusions which are tentative, to acknowledge the many variables involved in classroom learning. Design-Based Research aims to support action from research and so it is part of my research design that such ‘fuzzy’ or ‘tentative generalisations’ will be included in the outcomes of the case studies, which can then be tested against other cases and by readers in other contexts.
69