No significant differences were found between groups in terms of their performance on the discourse completion task (p=.19). This was disappointing, and unexpected, since the experimental treatment included materials and tasks specifically designed to develop learnersÕ pragmatic competence (see table 4.6 & appendix XI). There are at least three possible explanations for this lack of difference.
5.7.1 Hypothesis 1: A problem with the testing conditions?
Talking to a computer in the draughty corridors of a university, whilst being tape- recorded, doesnÕt exactly create the perfect conditions for the elicitation of speech acts and there is evidence from the learnersÕ diaries that these testing conditions caused students a lot of stress (see also appendix IX):
RM: [DCT] I completely couldnÕt do my best in computer test because it was almost the last part of class
and many people were in the corridor. I couldnÕt hear well and everything was disturbing meÉ unfortunately.
MT: [DCT] First I did a test with computer. I donÕt like this test, actually. I got upset and forgot every
words for the moment. I didnÕt do good.
YK: Some people took speaking test [DCT]. It was quite hard too and I got so embarrassed when some
students past by me. He was like ÒWhatÕs going on??Ó
SF: [DCT] I think speaking to your computer is shameful a bit.
It is quite likely then, that the testing conditions prevented learners performing to the best of their abilities. However, since both control and experimental groups were tested under identical conditions, we would still expect to see a difference if one really existed (as was found with the, equally stressful, student-student role-plays). This, therefore, is unlikely to be the reason for the lack of statistically significant differences on the DCT.
5.7.2 Hypothesis 2: A problem with the instruments used to measure pragmatic competence?
The second possibility is that the discourse completion task was simply not sensitive enough to detect changes in the learnersÕ pragmatic competence. In an early attempt to design a reliable measure of pragmatic competence using a DCT, Cohen & Olshtain (1981: 130) concluded that their efforts had Ôproduced at best a crude measure of such
competenceÕ and, although the DCT used in this study benefited from lessons learnt from trialing earlier models, perhaps it remains a blunt instrument? Every attempt was,
however, taken to make the DCT a reliable measure of pragmatic competence: a) A multi-media elicitation task was used so that students were able to see the
context of the required speech act for each situation on the computer screen, at the same time as hearing the prompt. This was done to limit the extent to which respondents were required to fill in missing contextual details from their imaginations, which tends to distort the results.
b) Rather than asking students to select appropriate responses from multiple-choice answers, they were allowed to answer the prompts in their own words in an effort to produce a more sensitive measure.
c) Students were asked to produce oral, rather than written, responses to the
situational prompts and these were recorded and transcribed for rating purposes. This process proved to be considerably more time-consuming, but was felt worthwhile because past research suggests that oral responses are a more realistic indicator of pragmatic competence than written responses, which tend to be shorter and less detailed.
The DCT did, however, have several weaknesses, which could have accounted for a failure to detect differences in learnersÕ pragmatic competence. The native speaker raters were only given written transcriptions of learnersÕ utterances, rather than the taped responses themselves, which meant that all the paralinguistic details, such as tone of voice, pitch changes or loudness (which are an important component of affective speech) were lost and learners were rated only on the actual words they used.
Secondly, the DCT only focused on the use of a single speech act (requests) with same- status or higher-status individuals, rather than a range of different speech acts. ILP studies to date have found no speech communities which lack speech acts for requesting, suggesting, inviting, refusing, apologising, complaining, complimenting and thanking (Kasper & Schmidt 1996), so perhaps these could act as a basis for DCT design.
Thirdly, although the NS raters received quite detailed instructions and guidance on the rating procedure (see appendix IV), their scores for the pragmatic appropriateness of studentsÕ responses differed markedly at times. An estimate of the DCTÕs reliability was obtained by comparing the authorÕs post-course ratings with those of all other NS raters, using Pearson product-moment correlation coefficient. There was a positive correlation between the mean scores [r=.60, n=59, p<.0005] but a coefficient of determination (r2)
value of .36 means that the two variables shared only 36% of their variance, suggesting the DCT had low levels of reliability.
As a further estimate of the testÕs reliability, a native speakerÕs responses to the DCT were added in for the post-course rating as a control measure (the NS raters were not aware of this since all responses were rated blind). The raters did rate the NSÕs responses as the most appropriate, giving her a mean score of 4.35, well above the combined total mean of 3.1 for the students in the treatment groups. However, they sometimes varied dramatically in their ratings of the NSÕs responses: in some scenarios giving a score between 1 and 5 for the same utterance. Scenario 5 of the DCT displayed particularly wide variations so it might be worthwhile examining this in more detail to investigate the reasons for disparity in ratersÕ scores. The DCT prompt for scenario 5 is shown below, along with the NSÕs response: