As discussed above, real-life academic writing tasks almost always involve some external reading materials. Such an integrated task setting, however, seems to be under-represented in most writing tests. Horowitz (1986a, 1986b, 1991) argued that there is a fundamental discrepancy regarding the use of primary or secondary reading materials between real-life tasks and most writing test tasks. Studies have been conducted to review the task types used in writing assessments (see Weigle, 2002, Shaw & Weir, 2007; Weir et al., 2013). Their results showed that the independent writing-only task type has played a dominant role in most high-stakes language tests and university admission tests. The independent writing-only task type refers to tasks which do not require the use of reading sources. Test takers are expected to produce the text by drawing solely on their internal resources, e.g. background knowledge on the topic. Among different genres, the essay task (i.e. test takers write an essay in response to a point of view, problem, or an argument provided in a single line prompt) is found to be very common in writing assessments. IELTS Writing Task 2 would be a typical example of an independent writing-only task (see below for the task).
(taken from IELTS sample paper
27
As the independent writing-only task type has served in many high-stakes language tests, there is extensive discussion about the issues of using such a task type in the literature. Following an improved understanding of the nature of academic writing ability, researchers (Moore & Morton, 1999, 2005; Moore, Morton, & Price, 2010; Plakans, 2008; Shaw & Weir, 2007; Weigle, 2002) argued that writing-only tasks might not be the most suitable tool to assessing academic writing ability. The two key issues are related to task authenticity and test fairness, which are discussed below.
2.3.1 Unsatisfactory task situational authenticity (context validity) Under the influence of the communicative testing approach, testing practitioners and researchers have become more aware of the importance of having test tasks which simulate reasonably the target language use context (Weigle, 2002; Weir, 1993; Shaw & Weir, 2007). Therefore, the design of the independent writing-only task type has been criticised for its lack of authenticity (see Cumming, 1997; Hamp-Lyons & Kroll, 1996; Lumley, 2005; Weigle, 2002, 2004). Writing tests, by nature, elicit sample performance from the test taker by using a very limited number of writing tasks. While it is impossible to simulate all real-life situations in any testing context, task authenticity is a fundamental concern of good language tests (Bachman & Palmer, 1996). Task authenticity can be achieved in terms of situational and interactional authenticity (Bachman & Palmer, 1996; Shaw & Weir, 2007). Shaw & Weir defined the situational authenticity as 'the contextual requirements of the tasks (2007: 9)' and the interactional authenticity as 'the cognitive activities of the test taker in performing the test task (ibid)'. In more recent frameworks of test validation (e.g. Weir, 2005), situational authenticity is part of context validity whereas interactional authenticity is part of cognitive validity (Test validation will be discussed in Section 2.5).
Situational authenticity (context validity) considers whether the test task itself is similar to the real-life tasks that the test takers are expected to encounter in the target language use context. As presented earlier, many studies which surveyed the writing demands in different academic contexts have concluded that academic writing is rarely done in isolation, but is overwhelmingly done
28
in response to source texts (e.g. Bridgeman & Carlson, 1983; Carson, 2001; Horowitz, 1986a, 1986b, 1991; Johns, 1981, 1993; Leki & Carson, 1994; Weir, 1983). Moore & Morton (1999, 2005) conducted one of the very few studies that compared test tasks and real-life tasks. They compared the IELTS Task 2 rubric with a corpus of 155 assignment tasks at both undergraduate and postgraduate levels across 79 academic departments in two Australian Universities. They made two specific observations regarding the discrepancy between the test tasks and real-life academic tasks in terms of the use of external sources and language functions.
First, the real-life academic tasks typically involved the use of primary sources (e.g. textbooks, journal articles, monographs) or secondary sources (e.g. a simple exhortation), either provided in tasks or collected by students. In contrast, they found that the test tasks did not engage test takers with external information. Test takers were required to process their prior knowledge while completing the task. Second, the real-life academic tasks usually involved more than a single language function. The most common functions were evaluation, description, summarisation comparison, explanation and recommendation. However, the predominant function identified in the test tasks was evaluation. The functions of summarisation and description were not identified in their sample of IELTS Task 2. Moore & Morton (1999, 2005), therefore, concluded that the typical essay test task seemed to represent the 'genre' but not specific contextual features of real-life academic essay tasks. Moore & Morton's studies have demonstrated the value of comparing the rubrics of actual test tasks to those of real-life tasks. This study will also compare the task features of reading-into-writing tests to those of real-life academic writing tasks. In addition to the rubric, this study will investigate the features of the reading materials of the reading-into-writing tests and the real- life tasks. The task analysis in Moore & Morton's studies was mainly conducted by the researchers. In this study, the tasks and the reading materials will be analysed by multiple sources (details are provided in Section 3.3).
29
2.3.2 Unsatisfactory task interactional authenticity (cognitive validity) Considering interactional authenticity, independent writing-only tasks require the test takers to engage in 'writing from internal resources'. However, the process of writing from sources is considerably more cognitively demanding than the process of writing from internal sources (Plakans & Gebril, 2009). Studies of task representation, discourse synthesis and summarising (e.g. Flower, 1990; Spivey & King, 1989; Spivey, 1990, 1997; van Dijk & Kintsch, 1983) reviewed previously showed that writing from sources required specific skills which may not be required when writers write from internal sources. Therefore, there is great concern about the construct being tested by the independent writing-only task type as it does not represent real-life academic writing processes. Another concern is that independent writing-only tasks would seem to encourage the 'knowledge telling' rather than 'knowledge transformation' approach to writing. Although knowledge telling is an approach typically employed by immature writers, Scardamalia & Bereiter (1987) argued that even advanced writers may use such an approach when they were asked to write on a 'knowledge telling' task, e.g. to produce a familiar genre which mainly involves recalling internal resources on a familiar topic.
2.3.3 Background knowledge effect (test fairness)
Another concern with the use of independent writing-only tasks is related to test fairness due to heavy topic effect. As argued previously, the independent writing-only task type requires the test taker to write drawing upon internal resources from their long-term memory. Weigle (2004) argued that students' performances are likely to be influenced by topic effect imposed by tasks which provide no input. This has inevitably led to test fairness issues when the topic of the writing task favours some test takers or is biased against others. Brown, Hilgers & Marsella (1991) conducted a large study to investigate the impact of topic on 3452 students' writing performance in a standardised language test. Ten topic prompt sets were used. The results showed that various topic prompts led to significant differences in the scores. The researchers thus concluded that writer's background knowledge on the topic
30
was a variable affecting the quality of his/her writing performance. Hughes (2003) argued that writing tasks in general language tests should not require the test taker to demonstrate specific topic knowledge. Douglas (2000) argued that an appropriate level of disciplinary topic knowledge should be part of the construct of the English for specific purposes (ESP) tests. However, the issue is less straightforward for English for academic purposes (EAP) tests. Although EAP tests are one type of ESP tests, the majority of EAP tests are not discipline-specific. The testing population of most large-scale EAP tests consists of test takers from a wide range of academic disciplines. In other words, these test takers do not share the same disciplinary background knowledge. Therefore, using independent writing-only tasks which require test takers to draw upon their background knowledge on the topic to assess their academic writing ability may not be the most appropriate method.
This section has discussed the use of independent writing-only tasks in writing assessments and the issues arising from the practice. The next section reviews and discusses the use of integrated reading-into-writing tasks.