2002 2004 Productores Subsidio Productores Subsidio

There is widespread recognition of at least two major approaches to conducting research in the social sciences (Mackey and Gass, 2005). One, often termed deductive or positivist, proceeds top-down: the researcher has very specific questions or even hypotheses in mind and prepares structured instruments to obtain data precisely focused on these prior issues, using categories decided upon in advance. Another, often termed inductive or constructivist (including grounded theory), proceeds bottom-up: the researcher has a general area of interest and some broad research questions, but uses open, unstructured instruments, with little control of what issues emerge from the data, and categories are found from the data during analysis. In the positivist approach, the preconceived ideas of the researcher, based on the literature or theory, are paramount; in the constructivist approach, it is the voices and perspectives of the participants that are paramount and are explored.

Having made this distinction, however, it must be said that it is often found in applied linguistic research, especially in the areas of teacher cognition and strategy research, that a judicious combination of the two paradigms is found. Indeed, this is our stance. It has already been seen (in chapter 2) that we have research questions but not hypotheses, and those questions, while not highly specific, are nevertheless not completely general either, but target some expected areas of interest such as scoring criteria, rating scale, the rating/decision making process, and effects of rater

background. Our instruments also, as will appear in more detail below, exhibit a combination of the two approaches. In particular, the data gathered is primarily qualitative, and qualitative data gathering is strongly associated with the constructivist position. On the other hand, we also use some quantitative data arising from some closed questions asked in the general interview, and secondarily from the qualitative data gathering, in the form of counts of occurrences of coded material.

In order to address the research questions, then, both qualitative and quantitative methods were used, involving basically three instruments: A. a general interview ascertaining participants' background characteristics and their beliefs about writing related issues; B. think aloud (TA) reporting to ascertain their actual rating process, criteria and scales used, and so forth employed as they evaluated essays; C. semi- structured immediate retrospective interviews conducted with raters just after each essay was rated in order to elicit more of their process of writing assessment. There was also some use made of observation by the researcher of what the raters were doing while rating, and of what they wrote on student scripts.

As Lumley & Brown (2005) have argued, interviews are commonly used to elicit views of raters and candidates about both ‘test-taking’ and rating, as an alternative or supplement to verbal reports (in our case the latter). Indeed, the use of more than one instrument to tackle the same issues, and both qualitative and quantitative data gathering (termed 'mixed methods'), is more generally viewed as strengthening a study through triangulation (Mackey and Gass, 2005; Dörnyei, 2007).

The three main instruments were implemented in a way that exhibits a balance between the constructivist approach and the positivist. The general interview contained some structured closed questions, decided on by the researcher top down, but also contained

some open response options. The instructions for the TA were largely non-directive and the interview questions only semi-structured, so this allowed the teachers' categories and views to emerge. The data analysis was partly driven by coding based on what was found in the TA data, but also at various stages informed by taxonomies of potential categories or codes from prior studies.

There is another way in which the study fits more with the constructivist approach, which is its contextualisation. While the positivist approach believes that facts can be uncovered from research which are generalisable across many contexts, and may involve research in quite artificial conditions to uncover them, with participants who are claimed to represent large populations of people, constructivists tend to believe that all knowledge is specific to particular real contexts, and specific individual people, as these are all different. The current study clearly takes the latter view. This study is designed to investigate a real course environment, where actual teacher raters evaluate their own students’ compositions. The teachers responded in their own individual ways to texts from current students in their classes (papers written for assignments which the instructors had not created and assigned only for the research) while sitting in their own offices, homes, or other places in which they normally worked. After evaluation, the students got back their scripts, as normal for writing done as a class assignment rather than an exam. Furthermore, we will not claim that what we find will necessarily be true of raters of ESL writing in other contexts.

As we saw in chapter 2, most past studies in our field in fact tended to create or assume a more artificial and test-like response environment for their research, such as that of Cumming (2002) who studied the decision making behavior of experience raters using previous TOEFL samples rather than their own students' compositions. Weigle (1994), who investigated the effect of training on experienced and inexperienced raters of ESL

placement compositions in a university context, again used a previous subset of ESLPE3 compositions written by university students.