ELEMENTOS QUE COMPONEN LA INSTALACIÓN Partes de la instalación:

In any research study the reader must judge the degree to which the research is both valid and reliable. In essence they must be convinced that the project has trustworthiness (Gillham, 2000). But the concepts of validity and reliability are more contentious for qualitative than for quantitative research. Indeed, the research community is still developing a consensus as to the appropriate criteria for assessing validity and reliability in qualitative research (Merriam, 1998).

3.4.1 External Validity

External validity is the extent to which the findings of a given study can be generalised and applied to other situations. A criticism often levelled at case study research is that it lacks external validity since the results are not able to be generalised. Classical science research, based along positivist lines, has traditionally been aimed at finding generalisations based on large samples of data generated from carefully controlled experiments with only a very few variables (Gillham, 2000). Indeed, when most people think of ‘research evidence’ they think of this ‘scientific evidence’. Social scientists, on the other hand, generally study smaller samples and have more variables to deal with. Scientific generalisations, therefore, are often not appropriate for summarising social findings because of the sheer complexity of social events (Bassey, 1999). Generalisation from one group of people, or one institution, to another, may be compromised because there are too many elements that are specific to that group or institution (Gillham).

Adopting a case study methodology, however, does not preclude an interest in generalisations. Generalisations are “matters of judgement rather than calculation and the task of case study is to produce ordered reports of experience which invite judgement and offer evidence to which judgement can appeal” (Stenhouse 1988, p. 49). In other words, it is up to the reader to decide whether the researcher has presented enough evidence to support any general statements that might have been made.

Generalisations made in case study research are retrospective in nature (Stenhouse, 1988). They provide us with an analytical look from a historical perspective. We can

70 look back over the evidence and make generalisations about what has happened and, perhaps, what might happen in the future. To add to this discussion, Bassey (1999), leaning heavily on the concept of ‘fuzzy logic’, introduced the notion of ‘fuzzy propositions’ (more tentative) and ‘fuzzy’ generalisations (less tentative). Essentially a fuzzy generalisation asserts that something may happen without any measure of its probability. It is a qualified generalisation, carrying the idea of possibility, but not certainty. Instead of ‘do x, instead of y, and your pupils will learn more’ we have ‘do x, instead of y, and your pupils may learn more” (Bassey, 1999, p. 46, emphasis in original). This viewpoint is consistent with Berg’s (2004) argument that, if case studies are undertaken properly, they should not only fit the specific individual, group or event studied, but also generally provide understanding about similar individual, groups and events.

To add to our understanding of the nature of generalisations made from case study research, Stake (1995) introduced the hierarchical terms ‘petites’ generalisationsand ‘grandes’ generalisations. The first—petites generalisations—refers to general statements made within a study and are relevant only to that study. The latter— grandes generalisations—concerns generalisations about issues with reference to the wider population, of which the case is one example. Stake concludes by reminding us that, above all else, it is important that researchers make clear the speculative and tentative nature of their assertions.

To increase the validity of generalisations, Tripp (1985) argued for a cumulative process in bringing case studies together. In essence, this is what is done in arenas such as psychology or the law where case studies have historically been the prevalent research methodology. Tripp saw this process as a means to provide ‘qualitative generalisations’ in which individuals, meeting the facts of a new case, apply them to their knowledge of similar cases, and so develop a personal understanding. This view is consistent with Stake (1995) who used the term ‘generalisation’ to refer to the learning process through which we individually acquire concepts and information and steadily generalise them to other situations as we learn more. Accordingly, through careful replication of case studies over time, we should be able to extend our ‘petite’ generalisations to ‘grandes’ generalisations.

71

3.4.2 Internal validity

Internal validity addresses the match between research findings and reality (Merriam, 1998). Is the researcher’s analysis and report of the evidence a ‘true and correct’ representation of respondents’ perceptions? To address this issue we must first consider two presuppositions. Firstly, that a true and unique interpretation of the data collected exists, and secondly, that this interpretation may be determined by standard applicable technical procedures.

If we consider the first presupposition, that a true and unique interpretation of the data exists, an interpretivist view of reality argues that it is socially situated and, as such, multiple realities exist (Bassey, 1999). Merriam (1998) argues that if reality is viewed in this manner, internal validity is a definite strength of case study methodology. The critical issue is not the determination of one singular and absolute truth but rather the assessment of the relative plausibility of an interpretation when compared with alternative plausible interpretations (Yin, 2003). Is the researcher’s interpretation of the data ‘trustworthy’? “[T]he concern is with meaning, understanding and description, rather than discovering an objective truth” (Bourke, 2000, p. 66).

If we consider the second presupposition, since multiple realities exist, this implies that a standard applicable approach is invalid in analysing the data as it will not identify multiple realities. From an interpretivist perspective, meaning and interpretations are contextually and culturally grounded. As such, naturally occurring discourse, such as interview narrative and the interpretation of observational data, rely on the shared cultural understanding between the researcher and the case study participants. It is this shared cultural knowledge base that allows for the analysis of the interview material by a process that Mishler (1986) calls ‘expansion’. In this process, researchers bring together all the shared knowledge and information available to help with the interpretation of the narrative. To accomplish this expansion of meaning researchers must use their ‘best understanding’, making references to other material as well as to the shared knowledge between the participants. They must introduce factual material from other parts of the interview as well as from general knowledge of the situation. Not all statements can be taken at face value and must be judged within the context of the broader discussion. As Gillham (2000) argues:

72 “Interpreting research data is more than a matter of good intentions. It requires discipline and concentration to present a ‘true’ picture: anything that gets in the way of that threatens the validity of your research” (p. 25).

In the current research project, the accuracy of the interpretation of the data is enhanced by the researcher’s background. As the researcher, I was a secondary school teacher for 18 years and, in my last position, I was responsible for the implementation of NCEA throughout the school. Additionally, I was a facilitator involved in the professional development of teachers during the implementation of NCEA. This background provided me with an in-depth knowledge of the philosophical underpinnings, as well as the organisational structure, inherent in the NCEA. As such, I am closer to the research topic through shared understandings with the participants.

3.4.3 Reliability

Reliability has traditionally been concerned with the ability to replicate a given research project; if the study was repeated would the same results occur? This perspective is based on the assumption that there is a single reality and that studying it repeatedly will yield the same results. However this traditional view of reliability has been identified as a difficult area in social sciences research due to the fluid nature of human behaviour and perceptions. Since individuals’ experiences will influence their interpretations of reality, there are no benchmarks to judge reliability in the traditional sense of quantitative research (Merriam, 1998).

Lincoln and Guba (2000) suggest that rather than using the term ‘reliability’ in the traditional sense it would be more beneficial to consider the ‘dependability’ or ‘consistency’ of the results obtained. Rather than requiring that an outside researcher obtain the same results by replicating the study it is more important that, given the data collected, the results make sense within the context of the research situation.

Within this research project there are two main reliability issues: how consistent the data from different participants was, and how likely is it that similar data and conclusions would emerge from replication with other samples of teachers or students

73 in other schools? Considering the first point, the researcher must be cognisant of the variability within the data, and report the consistency or contrasting nature of the data as reliably as possible. Recognition is also made that student’ perceptions are likely to develop over the course of the research project, due partly to their involvement in the research and partly due to their expanding experience base. As such the researcher must reliably report the consistency or changing nature of individual student’s perception over time.

The second reliability issue relates to whether similar data and findings would emerge from studying a different sample of students and teachers. The choice of case study methodology recognises that perceptions and experiences are contextually situated. As such, the data and conclusions generated from this research project are specific to the research participants and school. Secondary schools in New Zealand are legislated to respond and adapt to their community of learners, and the wider community within which the school is situated. Accordingly, it is reasonable to expect a range of teachers’ assessment practices and students’ perceptions of these practices. As such, it is acknowledged that the findings of this project represent a subset of the range of possible perceptions of secondary school students and teachers. Further research will need to be undertaken to more fully describe the complexity of the interplay between teachers’ practices and students’ perceptions’ regarding assessment experiences. Additional issues regarding the reliability of the various methods used in this research are discussed in subsequent sections.

3.4.4 Triangulation

Data triangulation involves the use of two or more methods of data generation with the aim of explaining the richness and complexity of human behaviour by examining it from more than one perspective. Reliance on a single method of data generation may bias, or distort, the researcher’s picture of events and relationships. However, triangulation should not simply combine different kinds of data, it should attempt to relate them, so as to counteract the threats to validity identified in each (Cohen et al., 2000). The use of multiple sources of evidence, each with its own strengths and weaknesses, is a key characteristic of case study research (Gillham, 2000).

74 Traditionally, data generation in case study research has centred on the qualitative techniques of interviewing, observation and document analysis (Merriam, 1998). However case study methodology does not preclude the use of quantitative methods such as the Assessment Experience Questionnaire (AEQ, Gibbs & Simpson, 2003) used in the current research. The AEQ will be discussed in more detail in a later section, but it is pertinent to note a number of key points as they relate to qualitative data and triangulation. Firstly, the AEQ seeks a quantitative measure of students’ perceptions of influences on assessment. These perceptions are socially situated and, as such, are a qualitative dimension of their assessment experiences. Secondly, the quantitative measure from the AEQ can be used to compare the perceptions of the nine case study students with the larger school population of Y12 Traditional Mathematics students. If it can be established that there is no significant difference between the two groups, then this increases the trustworthiness of the data generated from the case study students as being representative of the Y12 Trad maths cohort. In this respect, the AEQ serves as a valuable source of triangulation.

This research project is interested in the ‘realities’ students have constructed and how they make sense of the world they live in; realities that are constructed by individuals interacting with their social worlds (Merriam, 1998). From a sociocultural perspective these realities are personal, holistic, multidimensional, and ever-changing. With respect to this project, students’ perceptions of the formative use of assessment information are determined by their individual and collective experiences. Socially situated in time and space, students’ perceptions of reality will be fluid, rather than static. As such, we could reasonably expect to have a degree of variability in perceptions, both between students and within students over time and social and cognitive contexts. In accordance with this view, triangulation not only includes generating data using different methods, it also involves using the same method over a period of time. Accordingly, this project generated data over a school year.