Hortalizas

This sub-section reviews the relevant studies on criterion-related validity in the literature and highlights a few issues related to research nature and sample size, results of correlations between overall test scores and academic outcomes, and results of correlations between writing test scores and academic outcomes. Research nature and sample size

Previous studies of criterion-related validity between test scores and academic outcomes are predominantly quantitative. Most studies measure academic achievement by Grade Point Averages (GPA) (e.g. Allwright & Banerjee, 1997; Archibald, 2001; Avdi, 2011; Cotton & Conrow, 1998; Dooey & Oliver, 2002; Dooey, 1999; Feast, 2002; Green, 2005; Humphreys et al., 2012; Read & Hayes, 2003). Some used coursework grades (e.g. Ingram & Bayliss, 2007; Ushioda & Harsch, 2011). The sample size of the criterion-related validity studies varies widely from 17 (Read & Hayes, 2003) to thousands (Cho &

76

Bridgeman, 2012). However, most studies had a sample size less than a hundred.

Results of correlations between overall test scores and academic outcomes The findings of the correlations between overall test scores and academic outcomes in the literature are contradictory. Some studies showed no significant correlations between the two measurements. Cotton & Conrow (1998), who studied 33 students, found no significant correlations between the participants' IELTS bands and the language difficulties they experienced in their course work. Ingram & Bayliss (2007) studied 28 non-native students' language behaviour by students' self-assessment, interview, teachers' rating and researcher's observation. They found that while the students were generally able to produce the target language, there was no significant relationship between their IELTS scores and course-related task scores. Other studies such as Kerstjens & Nery (2000) shared similar results. They investigated the correlations between 113 first-year international students' overall IELTS bands and their GPA, and reported a non significant correlation coefficient of 0.15. Dooey (1999) investigated the university admission threshold level of IELTS 6.0 in particular. The results showed no evidence that students who did not meet such level are more likely to fail.

Some others found significant, but usually low, correlations between the two measurements. Criper & Davies (1988), one of the earliest studies, found a correlation of 0.3 between participants' English Language Testing Service (ELTS) scores and their GPA. Two decades later, Feast (2002) similarly found a significant but weak regression coefficient of 0.39 for between IELTS scores and GPA, with 101 international students in an Australian university. Cho and Bridgeman (2012), in their study of 2594 students, also found a weak correlation between TOEFL iBT and GPA. Their study was one of the largest scale studies of this kind. Yen & Kuzma (2009) investigated 77 undergraduates at a British University and found that their IELTS scores correlated significantly with their first semester GPA 0.46 and their second semester GPA at 0.25.

77

Results of correlations between writing test scores and academic outcomes

On the other hand, some studies compared the predictive power of the individual macro-skills, i.e. reading, listening, speaking and listening. Generally speaking, receptive reading and listening test scores tend to have better correlations to academic outcomes than productive speaking and writing test scores. Writing test scores tend to have lower correlations to academic outcomes; that could also be because speaking and writing tests usually use a more limited point-scale than reading and listening tests. A limited range of scores has been seen as one of the problems of criterion-related validity studies (Ingram & Bayliss, 2007).

Similar to the results of studies into the predictive power of overall test scores, the results of studies comparing writing test scores and academic outcomes are inconclusive. Cotton & Conrow (1998) found no significant correlation between 17 students' IELTS writing scores and their GPA. Humphreys et al. (2010), who studied 51 students from different disciplines, found no significant correlation between the participants' IELTS Writing test scores and their academic grades in either first or second term. Ingram & Bayliss (2007) investigated the relationships between students' IELTS writing scores and their self-rating regarding the difficulty they experienced in completing essays and reports. The findings revealed no correlation between the two measurements. However, studies with a larger sample size tend to report significant results. The earlier mentioned Kerstjens & Nery's (2000) study found a significant correlation of 0.25 between IELTS writing scores and GPA. Ushioda & Harsch (2011) found that IELTS Writing explained over 33% of the variance in academic coursework grades. While these results comparing writing test scores and academic outcomes are inconclusive, they have suggested the range of correlations found between writing tests and academic outcomes.

Implications from previous studies

While the above mentioned studies have provided insights into the relationships between test scores and academic outcomes, the use of GPA as a

78

measurement of academic outcomes has been criticised. Many researchers argue that final academic success arguably depends on a range of non- linguistic factors rather than solely on language proficiency (Davies & Criper, 1988; Ingram & Bayliss, 2007; Weir, 2005). Affective factors/motivation, learning strategy and social-cultural factors are the most commonly discussed factors in the literature (See for example, Cotton & Conrow, 1998; Ingram & Bayliss, 2007; Kerstjen & Nery, 2000). It is beyond the scope of this study to discuss these non-linguistic factors. However, the limitations of using GPA as the measurement of academic outcomes mean that it should be avoided. This study therefore will select more relevant external measurements of test-takers' real-life writing performance than using their GPA.

Secondly, previous studies tended to focus on the overall test scores, which are reasonable for their own research purposes, e.g. establishing the general degree of accuracy against external (final) measurements. However, information regarding individual papers is more useful for the purpose of test improvement. As reviewed above, only a limited number of studies investigated the correlations between writing test scores and academic outcomes. And none of them investigated the predictive power of writing test tasks which involve multiple verbal inputs or multiple verbal and non-verbal inputs.

While reading-into-writing tasks have been in use for a long time (see Section 2.3), the criterion-related validity evidence of reading-into-writing tests is scarce in the literature. The present research aims to examine the extent to which reading-into-writing tasks can provide predictive information about students' writing ability in a real-life academic context and if reading-into- writing tests would provide any new information which has not been revealed by writing-only tests.

79

2.6 Research Questions

In light of the literature reviewed in this chapter, the three main research questions to be addressed in this study are:

1. What are the contextual characteristics of the academic writing tasks that students would normally encounter in real life? To what extent do the reading-into-writing test tasks resemble these contextual features under test conditions?

2. What are the cognitive processes that students would normally employ to complete the real-life academic writing tasks? To what extent do the reading-into-writing test tasks elicit these cognitive processes from test takers?

3. To what extent can performances on the reading-into-writing tests predict test takers' ability to perform on real-life academic writing tasks?

Building on Weir's (2005) socio-cognitive framework, the present study attempts to establish the construct validity of integrated reading-into-writing tests by investigating their context, cognitive and criterion-related validity. This study first of all investigates the contextual features of real-life academic writing tasks and the cognitive processing employed to complete these tasks, in order to define the qualities of a valid task in assessing academic writing ability. It then examines the contextual features of two types of reading-into- writing test tasks (essay with multiple verbal inputs and essay with multiple verbal and non-verbal inputs) and the cognitive processes elicited by them. The results collected from both the real-life and test conditions are compared to provide empirical evidence of the contextual and cognitive validity of reading-into-writing tests in assessing academic writing ability.

In other words, RQ1 investigates whether the characteristics of the reading- into-writing tasks are an adequate and comprehensive representation of those that would be normally encountered in the real-life context. RQ2 investigates

80

whether the cognitive processes required to complete the reading-into-writing test tasks sufficiently resemble the cognitive processes which a test taker would normally employ in non-test conditions, i.e. are they construct relevant (Messick, 1989). Is the range of processes elicited by the test tasks sufficiently comprehensive to be considered representative of real-world behaviour, i.e. not just a small subset of those which might then give rise to fears about construct under-representation? Finally, comparisons are made between performances on the real-life tasks and the reading-into-writing tests to find out the extent to which performances on reading-into-writing tests predict test takers' ability to perform on real-life academic writing tasks (RQ3).

81

3 METHODOLOGY

3.1 Introduction

Firstly this chapter describes how the two academic writing tasks and the two reading-into-writing test tasks were selected as being representative of real-life academic and test tasks in this study, and describes the basic features of these tasks (Section 3.2). The chapter then describes the research design of this study with respect to the investigation of the three components of language test validity: context validity (Section 3.3), cognitive validity (Section 3.4) and criterion-related validity (Section 3.5). Each of these research design sub- sections presents the details of participants, data collection methods and instruments, data collection procedures and methods of data analysis (See Table 3.1 for an overview of the study). A summary of the chapter is provided in Section 3.6.

82

Table 3.1 Overview of the study

Focus Data collection in both real-life and test conditions Data analysis

Context validity (RQ1)

 Investigated the overall task setting by expert judgement (n=10) on 7 categories, i.e. purpose, topic domain, genre, cognitive demands, rhetorical functions, intended reader and knowledge of criteria

 Investigated the input text features by expert judgement (n=2) on 7 categories, i.e. input format, verbal input genre, non-verbal input, discourse mode, concreteness, textual organisation and cultural specificity, and automated textual analysis (17 indices measuring lexical complexity and syntactic complexity and degree of

cohesion)

 Descriptive analyses of the expert judgement results

 Non parametric independent sample tests to compare the

automated textual indices between a) real-life input texts and Test Task A (with multiple verbal inputs) input texts, and, b) between real-life input texts and Test Task B (with multiple verbal and non-verbal inputs) input texts

Cognitive validity (RQ2)

 Investigated the cognitive processing elicited in both the real-life and test conditions by a Cognitive Process Questionnaire (a total of 443 questionnaires were collected from 219 participants – 70 on real-life task A, 73 on real-life task B, 160 on reading-into-writing Test Task A, 140 on reading-into-writing Test Task B)

 Exploratory factor analyses to show the underlying structure of the cognitive processes elicited in each condition

 Non parametric independent sample tests to investigate whether the cognitive parameters distinguished the processes employed by high-achieving and low-achieving participants in each condition.

 Non parametric related-sample tests to investigate whether the cognitive processes elicited by the each of the two reading-into- writing tests resembled the processes employed by the test takers (whole population and in groups of proficiency) in the real-life conditions

Predictive validity (RQ3)

 Collected the participants' performances on real-life task A, real- life task B, two other real-life tasks (i.e. question-and-answer test and case-study exam) and the two reading-into-writing test tasks

 Descriptive analyses of the performances on each task.

 Analyses of the correlation between performances on the real-life tasks and Test Task A, and between the real-life tasks and Test Task B

83