2.2. MARCO CONCEPTUAL
2.2.6. Tipos de Alimentos
Data for the present study was collected in three comparable experiments. Materials used in the experiments are presented first, followed by descriptions of the participants, experimental procedures and the steps used in preparing the data for analysis.
4.4.1
Target L2 feature
The experiments employed a practice environment designed to provide L2 learners of Dutch with self-study oral grammar practice focusing on one specific aspect of Dutch word order, verb second (i.e. inversion of the subject and verb position when a phrase begins with another phrase constituent), which is known to be difficult for learners of Dutch (Jordens, 1988). Because deviations from this rule rarely cause communication breakdown, learners may remain unaware of their errors in spoken situations.
We designed exercises consisting of pairs of video clips and oral quizzes (see Figure 4.1). Learners first watch part of a story in the video clip and answer questions about the video by speaking their answers aloud. The oral quiz component is a word block ordering exercise: learners first read a question and then attempt to produce a grammatically correct utterance from the word blocks on the screen. More information on the exercise is available in (Penning de Vries et al., 2014b).
4.4.2
Automatic CF
Providing automatic CF in oral L2 practice requires the use of ASR to convert oral productions to text so they can be automatically evaluated for grammatical correctness, which is particularly challenging in the case of L2 learners (van Doremalen, Strik, & Cucchiarini, 2010). In the present system, a feedback
Figure 4.1: A screenshot of the Dutch word order exercise. The system presents learners with a video clip, after which learners answer questions about the clip by speaking Dutch. Learners read the question (1), mentally arrange the word blocks (2) into a correct order, and speak their answer aloud after pressing the record button (3).
module responds to recorded answers with CF that provides information on the correctness of the word block orderings.
If the answer is correct learners receive confirmatory feedback (Figure 4.2, left), otherwise one of two types of feedback is provided, depending on the system configuration. The first type is modeled after elicitation feedback (Lyster & Ranta, 1997). It signals the incorrect answer and prompts the learner to try again (Figure 4.2, centre). From their second mistake onwards, they also receive a hint whereby the next word block is automatically placed in the answer area. The second type of feedback is modeled after an explicit recast (Erlam & Loewen, 2010). The system displays a message notifying the learner of an error and displays the correct ordering of the blocks (Figure 4.2, right).
Figure 4.2: A screenshot of feedback in the system. A feedback module notifies the learner that their answer is either correct (left), or incorrect. Two types of CF are available: prompt (middle) or recast (right).
4.4.3
In-practice questionnaire
To tap into learners’ affective experiences during practice, we developed a short questionnaire designed to poll their affective states at regular intervals during practice (e.g. every 15 minutes). With such a design, it is important to ensure the questionnaire is minimally interruptive (Dela Rosa & Eskenazi, 2011).The questionnaire was therefore limited to three constructs, enjoyment, willingness to practice and self-efficacy, which were expected to be sensitive to the learning environment and CF. In line with situated perspectives on L2 motivation (Ushioda & D¨ornyei, 2012) highlighting the importance of taking the learning context into account, an effort was made to formulate questionnaire items specific to the CALL environment (cf. Dela Rosa & Eskenazi, 2011). Three five-point semantic differential items were used to survey in-practice affect (see Figure 4.3):
• Enjoyment: Oefenen met GREET is ... (Helemaal niet leuk - Heel leuk) Practising with GREET is ... (Not nice at all - Very nice)2
• Willingness to practice: Op dit moment wil ik ... (Iets anders doen - Verder oefenen met GREET)
At this moment I want to ... (Do something else - Continue practising with GREET)
• Self-efficacy: De volgende vragen zullen ... zijn voor mij (Moelijk [sic] - Makkelijk)
The next questions will be ... for me (Difficult - Easy)
4.4.4
L2 proficiency tests
To measure changes in learners’ V2 accuracy, we developed a computerised spoken Discourse Completion Test (DCT) (see Figure 4.4) consisting of 32 items, with 16 target items testing specifically for V2 and 16 filler items (Penning de Vries et al., 2014a, 2014b). Two different versions of the test, (A and B) containing linguistically equivalent, but not identical items were randomly assigned to participants in the pre-test and the post-test.
4.4.5
Post-practice questionnaire
To examine learners’ overall impressions of practice with the system, we developed a second questionnaire to be completed on the last day of the experiment, after the proficiency tests. Since in this case we did not have to worry about interrupting practicing learners, more items probing learners’ evaluation of different aspects of the practice were included. Two versions
2The first time this item was presented, the slightly different wording of Ik denk dat het
oefenen met GREET ... zal zijn (Helemaal niet leuk - Heel leuk). I think that the exercise with GREET will be ... to reflect that the user had not yet practiced with the system
Figure 4.3: A screenshot showing the in-practice questionnaire, with items for enjoyment, willingness to practice and self-efficacy (see text for translations).
were used: A CF version contained questions about learners’ experience of receiving feedback from the system, while a NOCF version contained questions on the importance learners attach to feedback. The two questionnaire versions shared seven items which we used for comparisons between the NOCF and CF conditions:
1. Ik vond het . . . om met GREET te werken (Saai-Leuk) I thought it was . . . to work with GREET (boring-fun)
2. Oefenen met GREET is een . . . manier om mijn Nederlands te verbeteren (Slechte-Goede)
Practicing with GREET is a . . .way to improve my Dutch (bad-good)
3. Mijn Nederlands is . . . geworden door te oefenen met GREET (Slechter- Beter)
My Dutch has become . . . as a result of practicing with GREET (worse- better)
4. De videos zijn . . . (Saai-Leuk) The videos were . . . (boring-fun)
Figure 4.4: A screenshot of the DCT. Each item consists of an unfinished Dutch sentence (left) together with a picture and hint to add context (right). Learners complete the item by formulating and then recording an answer before the time bar reaches its limit (bottom).
5. De vragen waren . . . (Saai-Leuk) The questions were . . . (boring-fun)
6. De videos waren . . . (Moeilijk-Makkelijk) The videos are . . . to understand (difficult-easy)
7. De vragen waren . . . (Moeilijk-Makkelijk) The questions were . . . (difficult-easy)
4.4.6
Participants
Adult learners of Dutch with proficiency levels A1 and A2 of the Common Euro- pean Framework (CEF) studying at language institutes near the university were recruited. These participants had a variety of L1s and cultural backgrounds. In total 31 participants were recruited for experiment 1, 30 for experiment 2, and 27 for experiment 3. In experiment 2 only, a group of German participants following an intensive Dutch course was included (N = 10) because of their low level of Dutch and prior experience learning English. Although German native speakers might be expected to be proficient with V2, some work shows that
learners of L3 German with prior exposure to English can have difficulty with V2 (Bohnacker, 2006).
4.4.7
Procedures
In experiment 1, learners visited the lab twice. In the first session, they completed a background questionnaire, two proficiency tests and a 45-minute practice session. Learners were randomly assigned to a CF-enabled condition and a CF-disabled (NOCF) condition and practiced for 45 minutes. Once before practice and every 15 minutes thereafter, the system paused the exercise and required the learners to fill out the in-practice questionnaire. In the second session, participants first completed practice, again filling out in-practice questionnaires, and then completed the proficiency tests followed by the post-practice questionnaire.
The experimental design for experiment 2 was identical to that of ex- periment 1, with one exception: learners were randomly assigned to either a CF condition providing prompt feedback (CF-P), or a CF-R condition providing recast feedback. All other aspects of the experiment were identical to experiment 1.
Experiment 3 was aimed at studying CF over a longer period of time. A total of seven 30-minute sessions were scheduled. Learners completed the background questionnaire and proficiency tests in the first session and the post-tests and post-practice questionnaire in the final session.
4.4.8
Data analysis procedures
Questionnaire data was extracted from the system database and learner proficiency was obtained by manually scoring the pre-test and post-test recorded utterances for word order correctness. Learning gain was calculated by subtracting the pre-test score from the post-test score. Together, the three experiments yielded data for 88 participants. This data was reviewed to remove participants who had technical difficulties (2), missed practice sessions (7), or had high pre-test scores indicating they had little to learn from the experiment (23)3. Excluding these participants left a total of 56 learners for the study.
Due to the low number of participants in each experiment group after filtering, it was decided to pool the data from the three experiments. To enable the pooling, we decided to compare the in-practice questionnaire data for the first and last practice sessions of each experiment. This was done because the third experiment included a different number of practice sessions and in-practice questionnaires compared to the first two experiments. The mean of the in-practice questionnaire data recorded during the first and last practice sessions provided two data points for each participant.
3High pre-test scores indicate mastery of V2 inversion and complicate the analysis with
ceiling effects. Inspecting background data for the 23 learners revealed that most were indeed native speakers of German (10) or had learned it as an L2 (7).