After review by some teachers and modification, the passage of Reading Comprehension was initially considered appropriate in length for the second language students as again judged by their teachers in Aceh, and the topic was known to the students through previous teaching. Some teachers in Aceh reviewed the Reading Comprehension passage (and the items) and they suggested decreasing the difficulty of the items and the English expression and vocabulary. In doing this, it may be that the items were made too similar.
The original Reading Comprehension test was piloted with sixty students who were either bilinguals or monolinguals. Opinions and suggestions were requested from the students and their teachers. The Reading Comprehension test then was redesigned by taking into consideration the students’ and the teachers’ opinions. Considered as difficult, the Reading Comprehension test passage was redesigned to contain less unfamiliar vocabulary, and pictures were. This led to the reconstruction of the test with more pictures and repetitive words. Feedback and opinions of students and teachers who preferred more pictures on the reading passage due to its benefit in understanding
reading were taken into account. With pictures, the passage became more accessible for the students because the pictures imposed clearer meaning on the content/passage. With repetitive words, the passage became more understandable for the students who
appeared to have limited vocabulary. However, by redesigning the test, it seemed that the items were made too similar, which led to the test not having enough difficult items.
It appeared that this redesigned Reading Comprehension passage was not so easy for some students and they could not answer the items as predicted. Some students with a medium-ability could answer the medium items correctly; some had difficulty with the predicted easy items. Similarly, it was predicted that the higher-ability students would be able to answer the easy items and medium difficulty items correctly, but they were unable to do so for all items. This resulted in a less than ideal agreement among the 780 students about the item difficulties along the scale. One reason for the
126 students still found the items difficult, even though the test had been redesigned prior to the actual test, after the piloting stage, and they guessed the answers to the items that they found difficult.
The presumed guessing that some of the students did is understandable due to their lack of English ability. They had only been learning English as a foreign language for about 30-40 hours in the semester when the test was conducted. Though the
curriculum stated that their ability was in line with the Reading Comprehension being tested, much of the time the teaching-learning progress ran much slower than the curriculum suggested. Teachers who were interviewed regarding this matter confessed that they could not run the teaching-learning process, as stated by the curriculum timeline because it would produce even poorer student achievement. The teachers, therefore, needed to pace the students’ progress, which meant that some students were behind the curriculum timeline. This could have resulted in students’ possessing very limited English when they sat for the Reading Comprehension test for this research. Their lack of ability in English could probably explain the question of why, after redesigning the Reading Comprehension passage test, with more repetitive words and pictures, the students still found the test difficult.
The construct validity of the test was tested by designing the items in ordered patterns of item difficulty which then were compared with their Rasch-measured item difficulties. That is, the Rasch measured item difficulty order given in Table 6.5 was similar to what was predicted when the items were created and designed. The overall fit to the Rasch Measurement Model, as shown by the item-trait interaction chi-square, was χ2
=200.4, df = 60 with p=0.000, which was not ideal. This meant that the agreement amongst all the students about the item difficulties along the scale was not ideal.
The Cronbach Alpha was 0.53 and low. However, some other measurement aspects were more satisfactory. The thresholds were ordered in line with the scoring categories and the Scoring Category Curves were appropriately ordered with overall measures. The Item Characteristic Curves were satisfactory in most cases, without being ideal, and targeting, while reasonable, would have been improved if there was a wider range in question difficulty. The student measures ranged from -0.8 to +3.9 logits and the item thresholds ranged from -1.2 to +2.6 logits.
127 Even though the fit to the Rasch Measurement model was not ideal, the Item Characteristic Curves showed no Differential Item Functioning against gender (male versus female) but the Item Characteristic Curves showed Differential Item Functioning against the type of teaching (bilingual versus monolingual teaching). The Differential Item Functioning (DIF) against type of teaching is interpreted as not really being DIF, but as showing that there is a real difference in output by bilingually-taught against monolingual-taught students, with Bilinguals achieving at a higher standard on most items. This is supported by the difference in overall Rasch measures for reading Comprehension by type of teaching. The bilingually-taught mean is 2.094 with N=394 and SD=0.85 and the monolingually-taught mean is 0.982 with N=386 and SD=1.07 (F=257.89, df=1,778, p=0.00000). These results showed that the Rasch measure was not ideal and still needed some improvement.
While girls performed significantly higher in English Reading Comprehension than boys overall — in line with the results from many western countries — only two items showed a statistically significant result for girls (item 10 and item 12). Item 10 was a multiple choice and it is hard to see why it should favour girls. Item 12 involved writing and English comprehension which possibly required more thinking and
motivation to write to answer the "why" question, and in retrospect, this could be expected to favour girls over boys, not because the item is biased, but because girls in their early secondary school years (12-13 years) were better than boys, in regard to English Reading Comprehension, as shown on items 1, 3, and 6 (see, for example Table 5.9 for Item 1).
In the case of language instruction (bilingual or monolingual instruction), five items showed DIF, with bilingual instruction being superior (items 2, 5, 6, and 9). Items 2, 5, 6, and 9 involved slightly harder reading words and comprehension than the other items and an understanding of their meaning would have been enhanced through bilingual instruction.
By performing DIF against both gender and language instruction, even with a less than ideal fit to the Rasch measurement model, the reason for the DIF could be worked out. Language of instruction would appear to be an important determinant of
128 performance in English Reading Comprehension and bilingual instruction was related to a superior overall performance compared to monolingual instruction.
The next chapter explains the experimental comparison results – pretest versus posttest measures by control and experimental groups for English Reading
129