La creencia en la causalidad

The literature on test taker factors that can affect performance on writing tests (see Barkaoui, 2007 for a meta-analysis of relevant literature) suggests several different test taker factors that should be

considered. The factors most commonly agreed on are language proficiency (Ruth & Murphy, 1984; Hoetker, 1992; Tavakoli, 2009), world knowledge (Ruth & Murphy, 1988; Read, 1990; Tavakoli, 2009); cultural and linguistic background (Ruth & Murphy, 1984; Hamp-Lyons, 1988; Johns, 1991; Kroll & Reid, 1994; He & Shi, 2008; Tavakoli, 2009) and educational background, including exposure to writing training (Ruth & Murphy, 1988; Johns, 1991).

2.4.1 Test taker factors – language proficiency

The language proficiency of the test taker can interact with prompt characteristics to have an effect on writing performance. The first interaction with a writing prompt is the act of reading and the reading proficiency of the test taker will partially determine the understanding of the prompt that is constructed by test takers. The more linguistically complex the language of the prompt, the more disadvantaged the weak L2 readers will be. Disparities in reading proficiency may cause misinterpretations of the prompt and create unexpected responses that may not be based on the assumptions of the prompt designer or the raters (Kroll & Reid, 1994). Misinterpretations of the prompt may lead to potential construct irrelevant variance (the effect of reading proficiency on writing test performance) and pose a threat to the validity of writing tests, especially for lower proficiency candidates.

The degree of rhetorical specification (linguistic and cognitive complexity) in the prompt and the

intended mode of response are two aspects of writing assessment that may interact with the proficiency level of test takers. The amount of rhetorical context that should be provided in writing prompts is a matter of debate. While there are arguments in support of full rhetorical specification, typically called

30

for to provide the test taker with an authentic context to write within, the amount of language that must be processed in this case may be overwhelming for lower proficiency test takers (Hoetker, 1982: 386). Ruth & Murphy (1984:419) argued that as writers mature and develop they may go beyond the rhetorical demands set by the prompt and want to demonstrate the wider discourse skills they have control of. Weaker test takers may be unable to accurately process a prompt with extensive rhetorical specification or a lengthy input text, possibly leading them to respond in a way not expected by the test maker. Conversely, high-proficiency test takers may wish to demonstrate their abilities by going beyond what the prompt asks for.

Another related issue is that student understanding of the intentions of the prompt may also override the expected mode (argumentative, expository or narrative) of the response (Hoetker, 1982). Hoetker claimed that the mode of writing any essay topic calls for is “precisely that mode that any particular student interprets it as calling for” (Hoetker, 1982: 379). This interpretation will be affected by the proficiency of the individual test taker. Response errors in terms of the specified mode may be due to misinterpretation of the prompt by weak readers, an inability to write in a more demanding mode (typically argumentative writing is seen as more difficult), or the desire of stronger test takers to write in a different mode to that specified in the prompt. Hoetker believed that some writers may not be

capable or willing to write in a particular mode even if that is called for by the prompt. Hoetker’s work was with first language, not ESL learners, so the extent to which his findings are generalizable to second language learners is a matter of conjecture. However, it seems possible that second language writers may be more likely to be susceptible to response errors as a result of misinterpreting the prompt than native speakers, either due to weak language resources in their L2 or because they are unfamiliar with the particular conventions of a specific mode of writing in a different language context.

The cognitive complexity of writing prompts and the response mode are two prompt characteristics that will be addressed in this work. As such, the research design must take into account and control for participant language proficiency. If the language proficiency of the sample population is not controlled for, the ability to interpret the relationships between the prompts and the test takers may be

jeopardized. For example, if the test takers who respond to one prompt are all high-proficiency language learners and the test takers who respond to another prompt are all low-proficiency language learners, the findings of any analysis of written product of responses from quite different populations will be attributable to the differences of the populations as much as any differences in the prompts.

2.4.2 Test taker factors – world knowledge and linguistic background

Ruth & Murphy (1984: 413) claimed that the meaning of any particular writing prompt depends on the “linguistic, cognitive, and social reverberations set off in the respondents. Both the language of the topic and the general knowledge of the participants interact in a writing test to determine what meanings the topic may elicit.” The intended meaning of the prompt may not be the same as the understood meaning of the test taker. The understood meaning will be dependent upon a range of test taker factors such as, “inadequate control of linguistic and semantic knowledge, weak commitment to succeeding on the test, inadequate world knowledge, and inexperience with testing contexts and conventions,” (Ruth &

31

subject that all potential test takers have enough relevant information on, or opinions of, to be able to write to the best of their ability. On the other hand the topic should not be too simple or predictable.” Early research in the field of topic effect suggested that familiarity with an essay topic advantaged those candidates and the depth of world knowledge a test taker brings is likely to be relevant to the sample of writing produced.

Test taker interviews performed with GRE candidates (Powers & Fowles, 1998) suggested that topic interest and topic knowledge were important variables in how test takers construct the difficulty of writing prompts. The test takers interviewed (Hispanic or African American test takers) consistently reported that they preferred prompts they could identify with, that drew on personal experience, that were clearly stated, that elicited strong feelings, and that were interesting (Powers & Fowles, 1988: 9). The sampled group described having most difficulty responding to prompts where they felt they lacked familiarity, knowledge or appropriate background along with prompts that were unclear or ambiguous, (p.11).

Research into the cultural and educational experiences of IELTS test takers indicated that Chinese and Greek students performed differently on the exam. Mayor, Hewings, North, Swann, & Coffin (2007) found that “low-scoring Chinese L1 candidates made significantly more grammatical errors than Greek L1 at the same level of performance,” (p.251). The study reported relatively minor differences in argument structure between the two groups but reported that “Chinese L1 writers also have a greater tendency to directly address the reader and to speak in the collective voice, (p.300).

ETS research reports on the TOEFL (see, for example, Breland, Lee, Najarian, & Muraki; 2004; Lee, Breland, & Muraki; 2004) consistently concluded that there are only very small statistically significant differences in difficulty across TOEFL writing prompts. TOEFL Research Report 76 (Breland, Lee, Najarian, & Muraki; 2004), which looked at writing prompt difficulty by gender found that there were statistically significant differences between male and female candidate performance on almost all writing prompts, but that the effect sizes were sufficiently small to be of little or no concern. The effect sizes were generally around 0.2 standard deviations of the overall score. While this does seem low, the fact that female candidates consistently scored better than expected and male candidates worse than expected, after controlling for language proficiency suggests that this may be an aspect of prompt design that is of potential concern and to be considered and screened for during pretesting. It also suggest that sample populations of studies into writing prompts should be balanced in terms of the gender of the

population.

TOEFL Research Report 77 (Lee, Breland, & Muraki; 2004) examined the comparability of TOEFL writing prompts for different first language groups. It focused on Asian language speakers versus European language speakers because of their differing performance overall on the TOEFL. The report concluded that there were no significant group effects between the two in the writing section after adjusting for overall language proficiency. However, in 27 of the 81 prompts analyzed there was “a statistically significant amount of variation in essay scores,” (p.14) as a result of native language effect. The study stated that the effect sizes were negligible to overall group performance but the fact that around one

32

third of TOEFL writing prompts resulted in first language background interfering with test takers’ true scores is troubling.

The studies reported in this section show that the makeup of the sample population is important when investigating complex relationships between test takers, writing prompts, and written products. The greater diversity there is in the sample population, in terms of linguistic background and world

knowledge, the less risk there will be that the results are influenced by confounding variables from the sample population. However, even diversity within the sample population may not completely guard against individual effects.

2.4.3 Test taker factors – cultural background

Test takers will not all necessarily come to an exam with the same cultural and linguistic frame of reference (Ruth & Murphy, 1984). This is especially true of international English language proficiency tests, which are taken by candidates from many different countries. These differing cultural

backgrounds can potentially cause different interpretations of writing prompts and lead to unexpected responses. An example given by Ruth and Murphy is that of a writing test that asks candidates to produce a “friendly letter.” Test takers who are unfamiliar with the expected conventions of the formal testing situation may produce a letter that is too friendly and hence be penalized for a misunderstanding of standardized testing conditions.

A similar point is made by Kroll & Reid (1994: 236), who stated that cultural interference can cause test takers to misconstrue or even miss the point of writing prompts. Those without the assumed cultural reference could be disadvantaged by certain writing prompts; especially those which assume a

knowledge of western cultural values. The authors cited the example of a prompt that asked test takers to respond to a topic about a “blind date.” Several ESL candidates interpreted this literally as asking about a date who could not see, leading to a number of unexpected responses.

Connor & Kramer (1995) reported that Korean, Belgian, Bolivian, and American students in a US business school interpreted and responded in different ways to writing assignments set in their program. Connor and Kramer concluded that the ESL students differed from native English speaking students in task representation. Task representation refers to the process of interpreting the writing prompt and then using it to organize a response. Interviews with the participants suggested that the reasons for the differing task representations were both cultural and educational. The Korean

participant differed most markedly from the task expectations and his difficulties were seen as typifying those faced by Asian language speakers when presented with academic writing tasks.

Johns (1991) explored in great detail the difficulties of an ESL student attempting to pass a standardized writing test. The Asian test taker reported having difficulty interpreting the instructions of the writing test. He also felt uncomfortable about the need for argumentation in the written response to the prompt. As a science student he was familiar with the demands of his field but was unsure of how to frame an opinion without scientific backing for an English writing assignment. These are clear examples of the types of pragmatic competence that Hamp-Lyons (1988) described as being potentially lacking in test takers unfamiliar with western academic traditions. The subject in Johns’ study described a lack of

33

knowledge of certain topics that appeared on the writing assessment he experienced. In these instances he reported being unable to focus on skills such as organization, as his linguistic resources were focused on overcoming his knowledge gap with the prompt topic. This test taker clearly felt he was

disadvantaged by some topics and his lack of background knowledge and awareness of academic expectations contributed to his poor test performance.

A more recent study found somewhat similar conclusions to Johns’ work. He & Shi (2008) interviewed 16 international students (from mainland China and Taiwan) who complained about culturally biased essay prompts and a lack of understanding of the requirements of the essay test. Several test takers described topics on plastic surgery, teenage crime, and divorce rates as requiring knowledge of their country of residence, Canada, which they felt they had little cultural knowledge of and as a result, little relevant language to respond with.

The findings of the literature reported in this section are based on qualitative methods; primarily interviews with test takers in the case of Johns (1991), Connor & Kramer (1995), and He & Shi (2008). The studies focus on how test takers engage with a writing assessment and detail the difficulties they face with a focus on cultural background and resulting lack of familiarity with standardized writing tests. These studies, although based on a very small number of test takers indicate that some individuals may respond to writing prompts in ways that are unforeseen by test designers. The ways that test takers engage with writing prompts and how they go about responding to them is an under researched area in second language writing assessment. The dynamic of the test environment and the interaction between the prompt characteristics and the test taker is a potentially rich source for gaining a better

understanding of how different writing prompts may have an effect on writing test performance and will be a key part of this thesis.

2.4.4 Other test taker factors

Age, gender, and ethnicity have also been reported as factors that can influence scores on writing tests. Gabrielson, Gordon, & Engelhard Jr. (1995) explored how a choice of writing prompts influenced

performance on a writing test for 11-th grade students in an American high school. 20 writing tasks were assigned to a total of 34,200 students in Georgia with the population being 52% female, 48% male, 67% white, and 33% black. The students were asked to complete a persuasive piece of writing within 90 minutes. Responses were analytically scored using four criteria: content and organization, style, conventions, and sentence formation. The main finding was that “the effects of the student characteristics on the essay scores are much greater than the effects of writing task,” (p.281).

The literature reviewed above (Sections 2.4.1-2.4.4) indicates that several test taker factors (linguistic, cultural, and educational background, along with other factors such as age, gender, and ethnicity) may have the potential to affect individual performance on a writing test. These factors will be considered when collecting written responses to be analyzed in this research. Efforts will be made to ensure that the sample population the responses are drawn from will be diverse in language background,

nationality, gender, and age. A sample population that is diverse in these factors will help minimize the confounding variables that may interfere with understanding the complex relationships between test

34

takers, prompts, and written products. The sample population that participated in this study is described in Section 4.4.1.