DETERMINACIÓN DE LA OXIDACIÓN DE LAS PROTEÍNAS

The design of the multiple case study in the present thesis is based on several considerations, all of them tied in one way or another to the definition of the object of study in Figure 3. I wanted to analyse the entire process of test development and validation, and thus it was necessary to ensure that both initial and operational test development would be covered by the cases. This would also enable me to analyse similarities and differences between initial and operational test development and validation. Since my basis of comparison is recommendations from theory, it made sense to select tests that would be likely to implement theoretical recommendations as carefully as possible. This would be the case with high-stakes tests, because these, if any, are likely to be held publicly accountable of their practices. And since theorists of test development and validation agree that purpose is the most important guideline for test development and validation, it was necessary that the tests I analyse serve the same purpose.

The design of the case study was further influenced by my interest in the role of the theoretical construct definition in test development and validation. I believe that from the perspective of theoretical design, the construct definition guides the questions asked and the procedures followed when test development decisions are made. Since theoretical rationale rather

than political decision-making practice is what I study in the present thesis, this is a central object in my study.

My belief in the significance of the theoretical construct definition is what Hamel et al. (1993:44) would call an initial theory of the present thesis. They define initial theory as “the initial idea that a researcher had of the perceived … issue or phenomenon” (Hamel et al. 1993:44). The initial theory influences the way that the object of study is defined, and it must be related to existing theories in the field being investigated. Since Hamel discusses case studies in sociology, he anchors the concept of initial theory to social issues and sociological frameworks. In the case of the present thesis, my belief or initial theory is related to language testing, educational measurement and applied linguistics. Its grounding in this research was presented in Part One of the thesis.

Although the concept of initial theory is closely related to that of the

working hypothesis in empirical research, I do not want to use the empirical research terminology. I will call my belief a belief or an initial theory, and consider it to lead to expectations about what the case analysis will show. The use of this terminology is by no means intended to downplay the defensibility or rigour of the case study as a research approach. I simply want to make it clear that the conceptual world of the present study is parallel but not equal to that of experimental research. Whereas working hypotheses are often pursued in experimental research to develop specific hypotheses that can be tested in future research, there is no such wish in the present study. Neither is the current design an ill-defined experiment. The design is intended to throw light on the complex processes of language test development and validation. The results are intended to specify the theoretical frameworks that the language testing research community uses for test development and validation so that they would help practical test developers develop quality into their tests.

For the purposes of the case study, I divide test development rationales into three possible categories according to the nature of the construct definition in them. In the first category, the theoretical construct definition is brief whereas the psychometric construct definition is detailed. In the second, the theoretical construct definition is extended whereas the psychometric construct definition is less extensive than in the first category. In the third, both the theoretical and the psychometric construct definitions are extended. I have chosen one case of reported practice to represent each of these three conditions.

Since the design in principle follows a two by two matrix where one dimension distinguishes between theoretical and psychometric construct

definitions and the other between brief and extended definitions, I must explain why I do not discuss four cases. The missing case is one where neither the theoretical nor the psychometric construct definition are extended. I do not discuss it because I do not believe this would be considered acceptable practice and because I believe it would not be likely that the development of such a test would be reported and published if it did exist. The test development cases that have been reported in the language testing literature do not represent perfection, of course, but those that have been reported through extensive reports do represent serious effort to meet quality requirements.

My initial theory and the case design lead to two main expectations. Both build on the assumption that emphasis on numerical construct definitions leads to a reliance on psychometric arguments to support the quality of the test while emphasis on the verbal construct definition leads to a reliance on theoretical quality criteria that concern test development rationales, validation studies, and the relationship between test development and validation. The first expectation is that the reports of test development will clearly reflect differences between the cases. The discussion in Chapter 2 showed that test development is a flexible and multi-dimensional process, and I expect that the type of quality standards used in a case will guide the questions asked, the materials investigated and the rationales presented for different test development decisions. The second expectation is that the association between the construct definition and the validation rationale will not be equally clear. As was discussed in Chapter 3, there are very few instructions and examples about how to use the theoretical construct definition in validation, so it is possible that all the cases show a concentration on psychometrically motivated validation designs.

If the initial theory is entirely wrong, the reports of test development and validation will only differ in terms of how they characterise the construct assessed, which was my basis for categorising the cases in the first place. If the analysis shows that there are clear differences in the reports but that the cases differ in many terms so that it cannot be said how far the nature of the construct definition can explain them, the initial theory was too narrow and a more detailed proposal can be developed. The initial theory cannot be confirmed, but support for it would be afforded if the expected differences in test development and validation procedures were revealed in the analysis.

The reason why I focus on the role and nature of the construct definition in test development and validation is my perception that the advice that theory provides is patchy and somewhat contradictory. Current validity

theory, in my interpretation, assumes that test development is guided by a theoretical construct definition. The centrality of the construct is not questioned; the discussion concerns how far into construct theory the test developers’ work should go. Somewhat in contrast, I suggest that current test development frameworks propose a distributed focus for development and validation, so that the construct is one criterion, but there are also other important concerns. Bachman and Palmer (1996), for instance, propose usefulness, while Alderson et al. (1995) and Weir (1993) propose validity, reliability, and practicality. Both validation and test development theory eschew the reporting of psychometric information only, with no theoretical elaboration of the construct. However, there are clear technical and methodological guidelines for how to provide psychometric information for tests. In contrast, guidelines for how to describe, discuss, and investigate the intended or realised construct on a theoretical/descriptive level are at a much earlier stage of development. If the theoretical construct definition is indeed as important as current validity theory states it is, test developers need advice and examples of its actual role in test development work.

In summary, the purpose of the multiple case study in the present thesis is to study the nature of reported practice in test development and validation, and especially to investigate the role of the verbal construct definition in it. The results show examples of different practices and enable comparison between theory and reported practice.