Análisis de la Demanda

Following our characterization of the general state of the art in NLP-based ICALL, the description of approaches to error diagnosis in the previous chapter and the intro- duction of the background for dialog modeling and dialog systems in the preceding part of this chapter, we now illustrate the current state of the art for ICALL systems that include some form of interaction and feedback by describing a selection of specific systems and their approaches. We begin with a general characterization of the require- ments and challenges that ICALL systems have to address, give a general overview, and briefly characterize commercially available systems before we detail the variety of research prototypes that have been developed.

Requirements and Challenges

Compared to the challenges for task-based dialog systems targeted at native speakers, interacting with learners comes with additional requirements. Learner language often contains particular errors that are more frequent and of different nature than errors of native speakers. These errors reflect the learning process. To recognize them, to incorporate them in the interpretation, and to provide corrective feedback is the main challenge of ICALL systems. Learner errors can make interpretation more difficult because they can increase the ambiguity. On the other hand, learner language is often simpler and more limited than native language. This can ease the task of language interpretation, since the language resources have to cover less. At the same time, it is also an additional design challenge to make sure that the vocabulary and syntactic structures used for system productions are appropriate for the targeted learner level. Giving feedback to learner language creates an additional thread in the dialog that has to be managed in relation to the content matter dialog thread.

Depending on the purpose of the dialog system, the content matter should be rel- evant and useful for the learner. In most cases, ICALL dialog systems will model a domain and task for the sake of practice, but in some cases, a real-purpose dialog sys- tem is adapted to non-native speakers (Raux and Eskenazi, 2004a,b)

42 CHAPTER 3. DIALOG FOR LANGUAGE LEARNING

Overview

The selection of the systems we present in this section is based on their relevance to the work undertaken in this thesis. We describe them under the aspects of input they expect from the learner and how this input is constrained, the error diagnosis and feedback they provide, the evaluation they have been subjected to, and the pedagogi- cal theories they were informed by.

In general, a large part of publications on ICALL applications concentrate on de- scriptions of the system and the interaction it allows but do not include any evaluation. If the systems are evaluated, this is often done in terms of their performance, i.e., the amount of errors they make (Seneff et al., 2004), or in terms of usability by means of questionnaires given to the users (Wang and Seneff, 2007; Lech and Smedt, 2006; John- son and Wu, 2008). Only a small number of ICALL publications include an evaluation of the language development or learning gains that their application can induce (Zhao, 2003). For an even smaller number of applications the learning gains are compared with alternative teaching means or across different parameters of the application. Re- lated to that, publications on ICALL applications only rarely make explicit reference to theories of second language acquisition. If they do, it is usually with the purpose to justify design decisions, but not in order to investigate the validity of specific SLA theories. We will point to exceptions to this rule below. We will first summarize the state of the art for off-the-shelf systems that are available to private users and then look in more detail at systems that have been developed within research contexts.

Off-the-shelf Systems

Commercially available ICALL applications usually focus on exercises related to new vocabulary and grammar rules. The learner input is constrained and systems are not geared towards free communication. If they contain any dialogic material it is used as a means to impart new language content, i.e., lexical items and grammatical struc- tures, similar to monologic lesson texts, rather than as a way to engage the learner in a conversation (e.g., “ActiveChinese” (Chiu, 2008) or “Side by Side Interactive” (Statan, 2006)). If called for at all, participation of learners is limited to advancing the presenta- tion of the dialog by clicking a button. Sometimes, learners can choose one out of a set of semantically equal options. In another variant of this task, learners can order a set of utterances to render a meaningful dialog. None of these systems allows free input to engage in a dialog. Some systems allow the user to record pronunciations of textual prompts and then give feedback about the quality of the pronunciation (Lafford, 2004; Chiu, 2008). In “Tell Me More” (Lafford, 2004) learners engage in a dialog by choos- ing an appropriate response from a set of three given candidates and then pronounce their response. The system’s speech recognition component then gives feedback about the quality of their attempt. Given that the learner input is very constrained, usually to multiple-choice questions, as described above, or fill-in-the-blank activities, the re- quirements for the the error diagnosis and feedback facilities are rather simple. In the simplest case, the system merely states whether or not the response was correct, which requires a simple comparison with the target response.

3.2. STATE OF THE ART IN EXISTING ICALL SYSTEMS 43 the telephone-based bus schedule information system Let’s Go in Pittsburgh (Raux and Eskenazi, 2004a,b). Although its original purpose and development was not geared to support language learning, it has been extended to cater for non-native speakers and it can give them implicit corrective feedback if their input deviates from the expected input. The application is meaning-based since its actual purpose is the real-life task of obtaining schedule information, but therefore its domain is very limited.

Since commercially developed off-the-shelf systems are rarely the subject of scien- tific publications, it is not surprising that there is a concomitant lack of evaluation of these systems, in particular regarding potential learning gains.

Unsurprisingly, there is a considerable gap between off-the-shelf systems and sys- tems developed in research contexts as described in the literature. In general, research prototypes provide more freedom in input, richer communication and more informa- tive feedback. However, since the systems are rarely accessible to the public, these claims are in general not verifiable. Research-driven systems are usually only available to a rather restricted number of learners, and in this context they are primarily used for the purposes of testing and further development. These systems can be roughly divided in those that support learning by providing distinct, often grammar-related exercises (Section 3.2.2) and those that support learning by engaging the learner in a dialog and meaning-based communicative interaction (Section 3.2.3). Although some systems include both aspects, one of them usually predominates.