This subsection examines the effect of lucky guessing on the part of speech section. Figure 2 illustrates the scatter plot of item difficulty and outfit t for this section. The horizontal axis shows item difficulty in logits, where larger numbers indicate more difficult items. The vertical axis shows outfit t whose values larger than 2.0 are taken as
81
misfitting to the Rasch model. This figure shows that eight items had values of outfit t > 2.0 and these items tended to be difficult. Figure 3 presents the scatter plot of person ability and outfit t. The horizontal axis shows person ability in logits, where larger numbers indicate more able persons. The vertical axis shows outfit t whose values larger than 2.0 are taken as misfitting to the Rasch model. This figure shows that low ability persons tend to be identified as misfitting.
Figure 4 illustrates the probability of success when a person with the ability Bn met an item with the difficulty Di. The horizontal axis shows the difference between person ability (Bn) and item difficulty (Di) for each response. A larger number in Bn-Di indicates a response resulting from a person with higher ability meeting an easier item. A smaller number in Bn-Di, on the other hand, indicates a response resulting from a person with lower ability meeting a more difficult item. The vertical axis shows the probability of a person with ability Bn succeeding on an item with difficulty Di. The smooth line represents the theoretical model. The model predicts that the larger the Bn-Di value is, the more likely it is that the person succeeds on the item, and vice versa. The dotted line, which represents the empirical data obtained from the participants, deviates increasingly from the expected model with smaller values of Bn-Di. In other
Figure 2. Item difficulty and outfit t for the part of speech section
Figure 3. Person ability and outfit t for the part of speech section
82
words, when people with low ability met difficult items, their success probabilities approached 25% (the expected percentage of correct responses by random guessing), which was higher than the model expectation.
Figures 2-4 may be taken to indicate that lucky guessing occurred when people with low ability met difficult items in the part of speech section. A close look at the response patterns by 13 participants with large outfit statistics (t > 2.0) also indicates the existence of lucky guessing. For example, Participant A (outfit t = 3.4) got the following difficult items correct despite a low person ability estimate (-1.26 logits):
a) She made the kind of excuse that people made at a big party when they wanted to densodate themselves from a conversation and move on to talk with another person. (difficulty = -0.2 logits)
b) From the 10th to 25th of October the show is held about various ways of having duterages such as tea and coffee. (difficulty = 1.62 logits)
c) The view was really beautiful as the light began to appear over the hills, and on the wide range of the sea; ahead, ascrice, and on either side of us. (difficulty = 2.65 logits)
83
On the other hand, this participant got the following easy items wrong:
d) When I was sitting on the bridge this afternoon, a big ship was passing, and I chonked my eyes from the sight of it. (difficulty = -2.2 logits) e) He watched her now as she famped the chicken over the fire. (difficulty
= -1.3 logits)
f) She had bought a new rotice for him. (difficulty = -1.1 logits)
The example a) was more difficult than the examples d) and e), perhaps because the test word followed to which might be mistaken as a preposition. The examples d) and e) were easy, perhaps because these items may represent the typical usage of transitive verbs: the test words followed the pronouns indicating a subjective case (I and she) and were followed by their objects (my eyes and the chicken). The example b) was more difficult than the example f), perhaps because the test word was used as an object of the gerund having. The example f) was easy, perhaps because the test word appeared in a short sentence, was marked with the article a, and was used in the familiar expression ‘buy something for someone’. The example c) was the most difficult item: the test word appeared in a long sentence and was an adverb without the typical -ly ending. Taken together, this participant may have relied on random guessing for getting difficult items such as examples a) - c) correct. The research design may allow such random guessing to occur, because a) no ‘Don’t know’ options were provided, b) the participants were asked to choose one option even if they had no idea about the item, and c) for validation purposes all the participants needed to work on items with varying levels of difficulty.
Lucky guessing was corrected by deleting response records which have difficulty greater than b + ln(m-1), where b is the person’s initial estimated ability and m is the number of choices (Wright & Stone, 1979). As each item had four choices, responses with item difficulty greater than b + 1.1 were deleted. This presupposes “that when items are so difficult that a person can do better by guessing than by trying, then such
84
items should not be used to estimate the person’s ability” (Wright & Stone, 1979, p. 188). As the result of this treatment, a total of 567 out of 8,547 (6.6%) responses were deleted and the number of items with outfit t > 2.0 decreased from eight to two (These two items will be inspected in Section 4.5.1).