La gestión de grupos El clima organizacional

Validity is one of the most important concepts when talking about assessment. It is at the center of all discussions of assessment (Fulcher & Davidson 2007; Fulcher 2010). In this section I will present what construct validity is, give a historical overview of how validity thinking in assessment has developed and look at validity in speaking tests in particular.

3.6.1 Defining construct validity

Until 1989 there had been used the same definition of construct validity for decades:

“By [construct]validity is meant the degree to which a test or examination measures what it purports to measure. Validity might also be expressed more simply as the 'worthwhileness' of an examination. For an examination to possess validity it is

necessary that the material actually included be of prime importance, that the questions sample widely among the essential over which complete mastery can reasonably be expected on the part of the pupils, and that proof can be brought forward that the test elements (questions) can be defended by arguments based on more than mere personal opinion.”

(Ruch 1924 quoted in Fulcher 2010) With this definition one can say that the important validation question was: does my test measure what I think it does? (Fulcher 2010). After Messick’s work from 1989 our

understanding of validity has changed. Now it is seen as “a single concept, with a number of different facets, or aspects” (Fulcher 2010, p. 20). After this validity is traditionally

understood to mean to find out if a test “measures accurately what it is intended to measure” (Hughes 1989 quoted in Fulcher & Davidson 2007, p. 4), or uncovering the “appropriateness of a given test or any of its component parts as a measure of what it is purposed to measure” (Henning 1987 quoted in Fulcher & Davidson 2007, p. 4). This in short terms means that validity is a question of whether a test measures what it is supposed to. For a more detailed introduction to the history of validity thinking, see appendix 3.

3.6.2 Validity in speaking tests

According to Luoma (2004) the question of validity is so fundamental in all assessment theory that it is very difficult to highlight aspects of validity that it more relevant to speaking tests than others. However, she has points of view on how to ensure a valid speaking test. She

34

says that “the validity of speaking scores is grounded in the purpose that the scores are intended to serve “ ( Luoma 2004, p. 185). Consequently, the first thing a speaking test developer needs to do, is to take some time to specify what is the purpose of the test. Further, the test developer should try to define what kind of speaking the test is supposed to assess, and what the construct is. When the purpose of the test is specified and the construct defined, the next step to ensure validity is to show evidence from the test development process that the test actually implements the construct (Luoma 2004).

To produce such evidence one has to use the task specifications to characterize the tasks, then explain the relevance of the tasks to the purpose and the degree to which they are representative of possible tasks this purpose can include (Luoma 2004, p. 186). The next step in ensuring valid speaking tests, is to evaluate the assessment criteria ( Luoma 2004). These have to be coherent with the purpose of the test, the construct definition and the tasks. They also have to be concretely defined so that they are easy to use. Finally, a step to ensure valid speaking tests is to look at the planning and monitoring the test developers do related to score use (Luoma 2004). One can look at administrative records, but it can also be relevant to look at the examinees experience with the test and look at the washback effect the test has (Luoma 2004).

3.6.3 Summing up

In the section about construct validity I have defined construct validity and looked at the development in validity theory. I have also looked closer at some theories concerning validity. In the last paragraphs I looked at validity in speaking tests in particular and presented steps to ensure valid speaking tests. In the following section I will give a chapter summary and briefly repeat what I have presented in this chapter.

3.7 Chapter summary

In this chapter I have looked at several theoretical aspects of assessment. I started the chapter by introducing what the Common European Framework of Reference for Languages (CEFR) defines as communicative competence. The reason why I did this is that the English subject curriculum in Norway is build upon the recommendations of the CEFR. Because the

35 what the curriculum is build on. Furthermore, I looked at the English subject curriculum. In doing so, I found that it is possible to argue that the competence aims that are relevant for the oral examination contain are so diverse aspects that it is possible to argue that there can be defined two constructs. The first construct that can be defined concerns the oral proficiency skills. These competence aims are found in the main area Communication in the English subject curriculum. The second construct that can be defined concerns content competence and these competence aims can be found in the main areas Culture, society and literature and Language learning.

After having presented the construct(s) to be tested in the oral examination, I moved on to looking at operationalizing of speaking tests, first by looking at task design, and next by looking at the development of rating scales. Finally, I presented the important concepts of reliability and construct validity. Reliability is in short terms a question of whether the scores of a test are consistent or not. In speaking tests there are three types of reliability that are especially relevant; intra-rater reliability, inter-rater reliability and parallel form reliability. Furthermore, the most common methods to ensure reliability in speaking tests, is the use of assessment criteria, rater training, the setting of cut scores and of course to make sure that the raters are aware of what might lower the reliability so that they focus on being consistent in their assessment. Validity is a question of whether or not a test tests what it is intended to test. To ensure the validity of a speaking test, there are several steps that can be taken; a speaking test developer needs to take time to consider what is the purpose of the test, define what type of speaking the test is to assess and also define what the construct is, then evidence from the test development process that the test actually implements the construct has to be shown.

In this last section I have tried to sum up what I have gone through in this chapter. In the following chapter I will present the methods I have used for my project before I present the results and findings of my project in chapter 5.

36

4 Methods

4.1 Introduction

In this chapter I will go through the procedures that have been used during the research process and the choices I have made while working on this project. First of all I will briefly go through an overview of what a research process consists of and then next I will go on to explaining my research process. I will go through the parts; preparations, collecting data, analyzing data, presentation of the data, and in the end I will discuss the validity of the results of my project.