Procedure
First, the students from both schools where English was bilingually-taught and where English was monolingually-taught answered, as pretests, a 12 item multiple-
95 choice reading comprehension test and two topics on which to write, and a separate attitude and behaviour questionnaire. Second, after a two-month time which was equal to 16 teaching-learning meetings, the same students sat for the identical tests, as posttests.
One research assistant was present on the days of the pretest and posttest administration at each classroom to ensure the student-respondents understood the instructions. The classroom teachers were allowed to be present on the days of the tests but were not allowed to interfere in the test process in order to maintain the validity of the students’ answers. The language of communication was the student-respondents’ native language (Bahasa Indonesian). The test took about 40 minutes to answer. After two months, the same students sat for an identical test, as a posttest, under the same conditions.
The written comments were taken from the posttest Questionnaire, not from the pretest Questionnaire, because it was assumed that the students could give answers better after they had studied English longer, in this case, after a two-month period. Out of the 780 students (bilinguals, N=394; monolinguals, N=386) who completed the Attitude/Behaviour Pretest, 702 students (90%) provided an additional written comment. There were no problems encountered in gathering these data.
All the students’ written comments, which were written in Bahasa Indonesia, were carefully read, translated into English, back-translated to ensure the accuracy of the translation, and classified into ten themes. Some student comments contained a number of themes. Each theme of a student’s written comment was entered into a Microsoft Word table in the form of tallies which then were counted and checked again to ensure that they were accurately recorded.
Pilot Studies
A pilot study was carried out in October and November 2010 in Aceh, Indonesia, regarding the English Achievement Tests, that is the English Reading Comprehension, the English Writing Test, and the Attitude and Behaviour
Questionnaire, on two types of students: bilinguals (N=31) and monolinguals (N= 28). The students who participated in the pilot study were excluded from participating in the main study that ran from January 2011 until April 2011. Some changes were made as a
96 result of the pilot study for three variables: English Reading Comprehension, English Writing, and Attitude and Behaviour.
Pilot Study on English Reading Comprehension
A pilot study was carried out regarding this test. There were three changes resulting from the pilot study. First, the content and wording were changed, although the title remained unchanged. It appeared that the majority of the students found the text hard to understand. It is assumed that this was due to the text containing both complex and compound sentences and unfamiliar vocabulary. Therefore, it was changed to text containing a small number of complex and compound sentences and to more familiar vocabulary. For example the sentences like, ‘Plants are called producers because they
have the ability to convert the Sun’s energy into chemicals that can be used to power life processes’ (Lines 4-5 of the original Reading Comprehension text) (see Appendix F
for the original Reading Comprehension Text) were found to be difficult, since they contained a compound sentence (word ‘and’), and complex sentence (word ‘that’). The sentence of the text was then changed to ‘All food chains begin with the sunlight. Plants
use the sunlight to grow’. The sentences were short, with repetitive words (sunlight)
(see Appendix A for the revised English Reading Comprehension test, which was the English Comprehension test for the main data collection for English Reading
Comprehension).
Second, a change was made to the pictures. Originally without pictures, the revised Reading Comprehension test contained colourful pictures. Pictures made the context clearer and more understandable for the students, especially for students with limited English in their initial year learning English as a foreign language. The
following is an example of change in term of pictures. Figure 4.2 is the original Reading Comprehension Test Front Page, and Figure 4.1 is the revised Reading Comprehension Test Front Page.
97 Figure 4.2 Original Reading Comprehension Test Front Page
98 Figure 4.3 Revised Reading Comprehension Test Front Page
99 The third change was related to time. The test was originally allocated 40
minutes but this was changed to 45 minutes. The original time was scheduled so that the Reading Comprehension Test and the Attitude and Behaviour Questionnaire were administered at one meeting of 80 minutes, consisting of 40 minutes for the Reading Comprehension and another 40 minutes for the Attitude and Behaviour Questionnaire. The Pilot Study suggested that students needed a little more time for the Reading Comprehension Test, while they needed less time for the Attitude and Behaviour Questionnaire. To accommodate the students’ need, the allotted time was changed for the main tests to become 45 minutes for the Reading Comprehension test and 35 minutes for the Attitude and Behaviour Questionnaire.
Pilot Study on English Writing
Changes were made to the topics. The original topics were ‘My Holiday’ and ‘My Hobby’. Due to the observations carried out during the Pilot Study and due to the English teachers’ comments in Aceh, the topics for the main data collection were changed to ‘My Idul Fitri Holiday’ and My Family’. There were two reasons behind this. First, it appeared that the students doing the Pilot Study did not have adequate vocabulary to answer the original topics sufficiently well. The students were in their initial semester of learning English and the Aceh teachers believed that the existing vocabulary learnt prior to the tests did not cover the original topics of the test.
Second, the pilot study showed that the majority of the students did not know what to write on the selected topics. This might mean that the context of the topics did not suit their daily context. In Acehnese, as well as in the Indonesian context, people do not have a specific time for holiday and hobbies, unlike in some other countries, like Australia, where people prepare and arrange their holidays during school holiday times. Therefore, due to the above reasons, the topics for the main data collection were
changed to topics that were believed to be better understood and suited to the Acehnese context. The revised first topic ‘My Idul Fitri Holiday’ is believed as a correct choice, since Idul Fitri is the Acehnese biggest religious celebration, which is similar to
Christmas in the Australian context. It is the most enjoyed time of the year. The second topic ‘My Family’ is well-known to the students, since they live with their family. Therefore, it was expected that the revised topics could give better opportunities to
100 students to express their beliefs and feelings, than the original ones (see Appendix C for the Revised English Writing test, which was the English Writing Test for the main data collection).
Pilot Study on Attitude and Behaviour with regard to Learning English
A pilot study was carried out regarding this test. The changes made were in three parts: (1) wordings of two statements; and (2) an additional category with three
additional questions. Firstly, it appeared that some wordings contained ambiguity for the students and, therefore, they were changed to avoid misunderstanding. The wording that were changed was under the category ‘Student/teacher relationship’ with the wordings originally “I like my English teacher teaching English reading”, revised to “I like the way my teacher teaches English Reading Comprehension”; and with the wording, originally, “I like my English teacher”, revised to “I like the way my teacher teaches English Writing”.
Secondly, answered essay questions provided by some students doing the pilot study of the Attitude and Behaviour questionnaire suggested to the author that further questions were needed for a better understanding of attitude and behaviour with regard to learning English. Therefore, a new category was added, called ‘Common Views’. It had three statements: “I like English because we use it in the classroom”, “I like English because it helps me in higher study”, and “I like English because it helps me go
abroad”. The full, original Attitude and Behaviour questionnaire is given in Appendix G, and the full revised Attitude and Behaviour questionnaire, which was the
Attitude/Behaviour test for the main data collection, is given in Appendix E.