The analysis involved running the data through the RUMM2030 program and checking for non-fitting items and items with poor discrimination, which were deleted, and then re-running the program. After four analyses, six of the original 18 items were rescored (Items 1, 2, 3, 10, 11, and 12); and another six (items 1, 4, 10, 13, 15, and 17) were deleted, leaving 12 items in the final scale: Items 2, 3, 5, 6, 7, 8, 9, 11, 12, 14, 16, 18. The six deleted items did not have a good fit to the measurement model, for two reasons. First, there were overlapping thresholds and poor Scoring Category Curves. Second, the item residuals were too high. Item residuals are differences between the predicted scores, based on the Rasch parameters, and the actual scores on the items. The RUMM program does not tell the researcher why an item does not fit the Rasch
Measurement Model but one can infer that many students with the same measures could not agree on the difficulty of the items all along the scale.
140 A more detailed reason for why some items did not fit the Rasch model was difficult to determine. However, it was assumed that a substantial cause of the misfit was the students’ lack of ability in the writing test. The students were in their first semester of the first year in which English was being taught for the first time. They were still weak in English in general and in English writing in particular. The misfit may have been primarily due to their low command of English writing related to the different classroom cultures (Indonesian’s way of writing is different from that of English’s way) in the two groups (bilingually-taught and monolingually-taught), in a test situation. Thus, it may be that the two groups of students (bilingually-taught and monolingually-taught) did not agree on the difficulty of the items.
The English writing test consisted of two topics. The first topic was ‘My Idul
Fitri Holiday’ and the second one was ‘My Family’. The difficulties of the items on the
two topics was predicted before data collection, based on logic and experience and the 18 items were placed on an ordered scale (see Tables 7.1 and 7.2). Paragraph
Organisation were ordered from easy to hard for Topic Sentence, Concluding Sentence and Supporting Sentences); Text Convention was ordered from easy to hard for
spelling, punctuation/capitalisation, and grammar); and Text Quality was ordered from easy to hard for readability, style, and text enjoyability (see Appendix D for the scoring rubric).
The three levels of difficulty were ordered from paragraph organisation (considered the easiest), to text conventions (considered to be hard), and text quality (expected to be harder). For Acehnese students learning English writing for the first time in their first semester of Middle Schools, it was considered that their previous knowledge and experience in Indonesian writing has helped them in understanding and ability in English writing. It was expected that the students would have acquired some paragraph organisation from their previous teaching and learning in Indonesian and, therefore, paragraph organisation was expected to be the easiest, because paragraph structure is similar in Bahasa Indonesia and English.
As students who have just started their English lessons, it was expected that they might find text conventions more difficult than paragraph organisation. The students would have had a small exposure to English and they not only lack depth in
141 vocabulary, but also in spelling, punctuation/capitalisation, and grammar. Thus, text conventions were considered to be harder than the paragraph organisation.
It was expected that students would find text quality the most difficult in their English writing and that they would need substantial English writing experience to do this well. Some students might be able to acquire paragraph organisation in which they are able to produce paragraphs with a topic sentence, some supporting sentences, and finish with a concluding paragraph. They might also be able to make their writing understandable by using the correct vocabulary, spelling, punctuation, capitalisation, and grammar. But, in the early stages, they generally find it difficult to ensure that the sentences are linked together so that the information ‘flows’ nicely for a reader, and that their writing has some style, so that it can be enjoyed by the reader . Therefore, the text quality was considered the most difficult of all.
Each topic (either My Idul Fitri Holiday or My Family), has an order of
difficulty of ‘easy’, ‘hard’ and, ‘harder’. For ‘Paragraph Organisation’, the ‘main topic’ was considered easy because it involved simple knowledge. The ‘Concluding Sentence’ was considered ‘harder’ because it involved a little more than simple knowledge. The ‘Supporting Sentences’ were considered ‘harder’ as they involved a higher order of thinking which students needed more practice in and knowledge of in order to write around seven or eight correct sentences.
For ‘text conventions’, ‘spelling’ was considered ‘easy’, because it involved common knowledge of the spelling of a noun, verb, adjective or adverb in English. In Indonesia, the spelling of word classes is among the first aspects taught.
‘Punctuation/capitalisation’ was considered ‘hard’ because it involved ability to recognise puntuation/capitalisation which might not be easy for students to do. ‘Grammar’ was considered the hardest compared with ‘spelling’ and
‘punctuation/capitalisation’, because it involved a higher order of ability and knowledge.
For ‘text quality’, ‘readability’ (i.e. clear and neat handwriting) was considered ‘easy’ because it involved a simple ability that students had already achieved. ‘Style’ (i.e. sentence fluency, for example varied length, good flow and rhythm, and varied structure) was considered ‘hard’ because it involved more ability and skills. ‘Text
142 enjoyability’ was considered the hardest of all under ‘text quality’ because it involved more advanced ability in order to make readers enjoy the writing.
Marks for the writing were allocated from the highest (4) to the lowest (0), based on the English writing scoring rubric (see Table 3.2). Each piece of writing was marked on the 9 items (for each topic) based on the English writing scoring rubric (see Table 3.2). Table 7.1 is an example of the marking used for Writing.
Table 7.1
Writing Marking Example for Topic 1 ‘My Idul Fitri Holiday’
Student
ID Item 1 (Title: My Idul Fitri Holiday)
Topic. Sent. Conc. Sents. Supp. Sents. Spelling Punc./
Cap. Grammar Read. Style. Text Enj. 001 3 1 4 2 2 2 3 3 2 002 1 1 4 3 3 2 3 3 2 003 3 2 4 3 3 2 3 3 3 004 1 2 4 3 3 2 3 3 3 005 3 1 4 2 2 2 3 2 2 006 3 2 4 3 3 2 2 3 2 007 3 2 4 3 3 2 3 3 2 008 3 3 4 3 2 3 3 2 3 009 3 1 4 3 3 2 3 3 2 010 3 1 4 3 3 2 3 2 2
The original behaviour of 18 items (item difficulties for both writing topics) for the difficulties prior to the Rasch analysis and after the Rasch analysis differs. This is due to the fact that the 18 item difficulties after the analysis were fitted to the Rasch model. Table 7.2 shows the original of 18 item difficulties prior to the analysis (with ‘Topic 1 My Idul Fitri Holiday, topic sentence the easiest, and Topic 2 My Family, text enjoyability, the most difficult).
143
Table 7.2
Initial Marking of English Writing Essay as ‘Items’ (Scored in terms of 18 Items for Compatibility with RUMM2030)
No. Original Number
1 1 Topic 1 ‘My Idul Fitri Holiday’ Topic Sentence 2 2 Topic 1 ‘My Idul Fitri Holiday’ Concluding Sentence 3 3 Topic 1 ‘My Idul Fitri Holiday’ Supporting Sentences 4 4 Topic 1 ‘My Idul Fitri Holiday’ Spelling
5 5 Topic 1 ‘My Idul Fitri Holiday’ Punctuation and Capitalisation 6 6 Topic 1 ‘My Idul Fitri Holiday’ Grammar
7 7 Topic 1 ‘My Idul Fitri Holiday’ Readability 8 8 Topic 1 ‘My Idul Fitri Holiday’ Style
9 9 Topic 1 ‘My Idul Fitri Holiday’ Text Enjoyability 10 10 Topic 2 ‘My Family’ Topic Sentence
11 11 Topic 2 ‘My Family’ Concluding Sentence 12 12 Topic 2 ‘My Family’ Supporting Sentences 13 13 Topic 2 ‘My Family’ Spelling
14 14 Topic 2 ‘My Family’ Punctuation and Capitalisation 15 15 Topic 2 ‘My Family’ Grammar
16 16 Topic 2 ‘My Family’ Readability 17 17 Topic 2 ‘My Family’ Style
18 18 Topic 2 ‘My Family’ Text Enjoyability
Note: The item difficulties for the Writing test have been ordered from Topic 1 ‘My Idul Fitri Holiday’
to Topic 2 ‘My Family’, and scoring is set out in terms of ‘items’ for compatibility with the RUMM2030 computer program.
Unlike Table 7.2 that shows the original of 18 item difficulties, before being analysed by RUMM230, Table 7.3 shows the 12 marked items whose data fit the Rasch measurement model, with Topic 1 My Idul Fitri Holiday concluding sentence as the easiest marked item and Topic 2 My Family text enjoyability as the most difficult marked item.
144
Table 7.3
Final Marking of English Writing Essay as Items (Scored in terms of 12 Items for Compatibility with RUMM2030)
No. Original Number
1 2 (easiest) Topic 1 ‘My Idul Fitri Holiday’ Concluding Sentence 2 3 Topic 1 ‘My Idul Fitri Holiday’ Supporting Sentences
3 5 Topic 1 ‘My Idul Fitri Holiday’ Punctuation and Capitalisation 4 6 Topic 1 ‘My Idul Fitri Holiday’ Grammar
5 7 Topic 1 ‘My Idul Fitri Holiday’ Readability 6 8 Topic 1 ‘My Idul Fitri Holiday’ Style
7 9 Topic 1 ‘My Idul Fitri Holiday’ Text Enjoyability 8 11 Topic 2 ‘My Family’ Concluding Sentence
9 12 Topic 2 ‘My Family’ Supporting Sentences
10 14 Topic 2 ‘My Family’ Punctuation and Capitalisation 11 16 Topic 2 ‘My Family’ Readability
12 18 (hardest) Topic 2 ‘My Family’ Text Enjoyability
Note. The misfitting ‘items’ are 1, 4, 10, 13, 15, and 17 and they were deleted.
The following the output from the RUMM2030 program supports the creation of a linear scale of Writing English with 12 ‘items’.