Unfortunately, a few issues took place during data collection and analysis that may influence the results to some extent. These issues concern the discrepancy between the lists of verb forms and the fact that some of the participating teachers did not fully follow the instructions they were given. However, while this is regrettable, luckily these issues do not affect the results in a serious way and remain marginal.
The list given to the teachers to rate differs slightly from the list of actual student production. Luckily, however, the effect the difference has on the results is minimal, and there are various reasons for why this happened. First, the list given to the teachers was drawn up before students who did not actually qualify were excluded; for this reason, there are a few forms that such students used in the teachers’ version, although no student in the study itself used such forms. Second, the students responded to the test and the questionnaire by hand. It was difficult to read some people's handwriting, but I have done my best to do the students justice. When potential spelling errors were spotted, the relevant letter or letters were always compared with other instances of the same letter to verify that they were not misinterpreted. In cases that remained unclear, the students were always presumed to have spelled the word correctly. However, if misspelling seemed evident, the word was recorded misspelled even in cases where the intended response would have been easy to guess, as in lalked for talked. This was done to be systematic in the coding. The student response sheets were examined twice, first for compiling the list for teachers and later for the actual analysis. Occasionally, if the handwriting was ambiguous, my decision on what letter(s) the students had used seems to have changed the second time. To verify transcriber and intra-coder reliability (e.g. Révész 2012, 216; Gass 2015), all lists, files and computations have been double-checked, some even triple-checked, after the original data entry, coding and analysis.
The third reason for some discrepancy between the student and teacher lists is that the computer program used (Microsoft Word) first kept autocorrecting some non-standard forms (e.g. did’nt was changed to didn’t automatically), despite attempts at switching off the automatic correction function. This means that some unconventional spelling provided by the students is missing from the teacher list but exists in the student list. Fourth, during the first round of listing, double answers (where a student gave two answers to a slot, e.g. am talking / was talking) were listed as two separate items (e.g. as am talking and was talking), while during the second round, double answers were listed as one unit, with the two forms separated with a slash (/). The same system was followed if the student’s response contained brackets (e.g. takes normally (a)). Fifth, the list from the second round also includes the cases where no answer was given. Sixth, some answers had accidentally not been included during the first round.
However, the above shortcomings have a minimal effect on the results. The vast majority of responses that were not included in the teachers’ list were given by one student only. This is 149 forms, of which 37 were forms where there was a double answer with either a slash, as in visited / had visited, or brackets, as in (had) had. There were also instances of no answer in 57 slots, with 1 to 44 students not supplying an answer, resulting in a total of 207 cases of no answer, and 9 forms with 2 students supplying the same answer that were not rated by the teachers. Similarly, of the verb forms listed on the teachers’ version but not included in the actual student responses, 150 out of 211 have been rated as inappropriate by all the 13 teachers, and in 32 cases, 12 teachers have rated the form as inappropriate. This leaves 29 cases where 11 teachers or fewer found the form inappropriate, but some teachers also found the form the best, acceptable or questionable. Since the number of these problematic cases is small and mainly affects individual students, and not systematically the same student, these accidental omissions do not bias the results. For a discussion on sources of error in coding, see e.g. Révész (2012, 204), who grants that it is “almost impossible to eliminate errors completely”.
Although the students were asked not to use any material to help them when completing the test (see Section 6.2), some were given the test as homework, and there is no guarantee that they did not, for example, consult a grammar book or discuss their choices with someone. However, as the students would gain nothing from disregarding the instructions for the test and as no student had a perfect score, it is very unlikely that they did so. In addition, one of the teachers chose to go through the test together with the students, asking them not to change their original answers after learning the suggested correct alternatives. Nonetheless, she was not able to control that this did not take place. Two students were disqualified from the study as in their
cases it was clear the responses had been altered (see Section 6.2). It is possible that some other students did so as well without this being as evident as in the abovementioned cases, but again, students would gain nothing from doing this and therefore it cannot be considered likely that they altered their responses. Intentional silliness (cf. Dörnyei 2007, 204) is unlikely as the students were no longer teenagers.
Unfortunately, the fact that teachers at Helsinki University of Technology did not keep the test sets given to students taking different courses separate may bias the results, because there were both students from mandatory and optional, remedial courses. However, the Technology group scored well, but it would have been interesting to know whether the students from remedial courses at the University of Helsinki and Helsinki University of Technology would have provided similar scores and whether the low-scoring students in the Technology group were from a remedial course.