Apresamiento del buque

Reading comprehension tests were employed to explore the effects of blending AOD with FTF instruction on students‘ comprehension achievement by measuring the differences between the control and experimental groups in reading comprehension test scores. This form of testing was applied to answer the first research question: do students who participate in an AOD as a supplement to FTF learning perform better on reading comprehension tests compared with students who engage only in FTF learning? And to answer the second research question: do students in each group (FTF and BL) demonstrate significant improvement in their reading comprehension scores from pre-test to post-test? The design and implementation of this test are described in the following sections.

4.3.1.1 Designing pre- and post-tests

A review of previous research in the Arabic context revealed that there were no existing reading comprehension tests to measure students‘ reading comprehension achievement at the first year of secondary school level that included the three levels of comprehension. Therefore, the pre- and post-test instruments were designed by the researcher, in consultation with four secondary school Arabic reading teachers and revised by a panel of

teaching reading experts – four university teachers in Saudi Arabia and three postgraduate students who had experience in teaching Arabic language in Saudi Arabia and who were completing their postgraduate studies in linguistics and education in Australian universities. The instruments were developed in stages, which included: (a) identifying the test details (objectives and number of terms); (b) creating an item matrix to detail content, form, timing and the scoring process; and (c) ensuring validity and reliability through the use of pilot tests (Cohen et al., 2000).

Each test included two texts and 17 questions. The four texts were selected from textbooks used in the first year of secondary school in Saudi Arabia. These textbooks and texts were designed and developed by the Ministry of Education of Saudi Arabia. This reading curriculum and texts are taught in all first year of public secondary schools in Saudi Arabia.

The reading comprehension tests were based predominately on Barrett‘s (1968) cognitive taxonomy of reading comprehension (as cited in Pearson & Johnson, 1972), but also referred to other existing comprehension taxonomies and categorisations (Ammar, 2009; DeBoer & Dallmann, 1970; Dechant, 1991; Dillon, 2007; Karlin, 1978; Morris & Stewart- Dore, 1986; Roe et al., 2010; Zintz, 1970) plus the objectives of teaching Arabic reading comprehension (Alshalan, 2008; Ministry of Education, 2008). The tests were designed to measure three main reading comprehension levels: (a) literal comprehension, referring to restating details and information that are clearly manifested in the text; (b) inferential comprehension, including inferring word meaning, understanding main and sub-ideas in the text, making inferences and drawing conclusions; and (c) evaluative comprehension, involving evaluating and analysing the author‘s purposes for writing the text and evidence used to support the author‘s ideas, author‘s feelings, and distinguishing between the facts and

opinions in the texts. The three main comprehension levels were sub-divided into the following strategies for the pre- and post-tests:

(a) Literal level (4 questions, 5 marks): restating literal information and details (4 questions, 5 marks)

(b) Inferential level (8 questions, 8 marks): inferring main and sub-ideas (2 questions, 2 marks); drawing conclusions (4 questions, 4 marks); inferring word meaning (2 questions, 2 marks).

(c) Evaluative level (5 questions, 5 marks): evaluating the purposes of writing the text (e.g., to inform, persuade or entertain) (1 question, 1 mark); evaluating the type of evidence used by the author (1 question, 1 mark), identifying the author‘s feeling in the text (1 questions, 1 marks); distinguishing between the facts and opinions in the text (2 questions, 2 marks).

Overall, each test included two types of questions: 15 multiple choice questions and two short answer questions. The total score was 18 marks. An answer sheet was developed for multiple choice questions and a rubric of possible short answers was also prepared. (See Appendices A and B for Arabic and English versions of the tests and Arabic version of the text passages.)

4.3.1.2 Piloting the tests

The research instruments were piloted to evaluate reliability and validity, observe the time to complete, and to explore potential problems respondents may face in terms of instrument wording or formatting (Colton & Covert, 2007). The processes of piloting reading tests were as follows.

4.3.1.2.1 Test validity

Validity is important and required for effective research (Cohen et al., 2000) and is a central aspect of research measurement (Neuman, 2006; Punch, 2005). Validity refers to the

extent to which tests measure what they aim and claim to measure (Cohen et al., 2000; Neuman, 2006; Punch, 2005). Content validity (Punch, 2005) was one of the main methods for validating the research instrument in this study; that is, the instrument must cover and measure the content and domains that are intended to be covered and assessed (Cohen et al., 2000). In this study, to achieve the content validity the professional and expert judgement method was applied (S. Chandra & Sharma, 2004; Cohen et al., 2000). The tests were given to a panel of 11 experts in teaching Arabic reading (four Arabic language teachers, four university teachers, and three postgraduate students) who provided written professional opinions and suggestions to improve the tests and ensure they measured students‘ reading comprehension. In addition, the expert panel evaluated the clarity, structure, language and comprehension levels of questions, and their validity in measuring these skills. Based on the experts‘ comments, the researcher amended the tests in terms of question order, adding or deleting some questions, and rewording some. For example, teachers on the expert panel suggested adding some questions that focused on identifying the authors‘ feeling and tone in the text. In addition, some questions were reworded to be more specific, clear and accurate, for example, correcting the Arabic grammatical structure of some questions.

4.3.1.2.2 Test reliability

Test reliability refers to the stability and consistency of tests and measurements (Neuman, 2006). A test-retest approach was used to measure the reliability of the tests and check if they produced similar results at different times (Cohen et al., 2000; Neuman, 2006). The same reading comprehension tests were administrated to 70 secondary school students in order to measure the internal consistency. Those 70 students who participated in the pilot tests were chosen from the same city where the main study was conducted. The interval time between test and re-test administration was two weeks, in line with the recommended optimal

interval between one to two weeks (Hartas, 2010). Subsequently, the researcher correlated test one and test two to evaluate reliability.

The results of the test-retest indicated a reliability coefficient of r = 0.86 for the pre- test and r = 0.84 for the post-test. This indicated that these tests were sufficiently reliable and consistent across time, as r > 0.7 is considered reliable and acceptable (Domino & Domino, 2006). Regarding the timing, the pilot tests established that it took students 40 to 45 minutes to complete either test.