La lenta recuperación de la industria juguetera

2.2

Natural language processing in ICALL

2.2.1 Motivation

While a large proportion of today’s CALL applications make no use of NLP tech- niques, the need and value of such an enhancement has been widely recognized (Meur- ers, 2012; Nagata, 2009; Heift and Schulze, 2007). The main argument for employing NLP is its ability to cope with relatively free and unconstrained learner input. Meurers explains the advantages of NLP in the following way: Traditional language activities such as, for instance, multiple choice questions or gap-filling involve only a small set of predefined learner responses and an equally small set of system responses. In such a context, learner responses and the corresponding feedback of the system can be enu- merated explicitly. Comparing the actual learner response with the set of expected responses is a matter of simple string comparison. However, this approach becomes unfeasible if the goal is to allow the learner to produce language freely, as in commu- nicative, meaning-based tasks. Also for more constrained activities such as summa- rizations or sentence translations, the number of possible correct answers is too large to be listed extensionally. This is because natural languages are rich and one meaning can be expressed by many different realizations. Nagata (2009) illustrates this problem by showing how a seven word target response for a translation task from English to Japanese can result in more than 6000 correct responses and almost a million possi- ble incorrect responses through the combinatorial explosion of lexical, orthographical and word order variants. Enumerating these variants and the corresponding feed- back extensionally is obviously not feasible. Therefore, a more concise, intensional representation of possible learner responses and the mapping to feedback is needed, if one wants to treat relatively free learner input. This can be realized using recursive structures or linguistically informed grammar formalisms instead of extensional list of strings (Meurers, 2012; Nagata, 2009; Heift and Schulze, 2007).

2.2.2 Expectations and challenges

Although the benefit of NLP techniques in CALL is commonly acknowledged, in- stances of NLP-enhanced ICALL are still rather rare within the greater field of CALL today. In a review of CALL literature, Stockwell (2007) mentions NLP only as a side note, the vast majority of the technology he reviewed does not use NLP techniques. Further, if NLP is used, it is often not very sophisticated: “most grammar programs are still very basic in the ways they process learner input, diagnose errors, and pro- vide feedback” (Levy, 2009, page 770). One reason for this may be the considerable cost and effort that is required to develop such NLP tools and resources (Schulze, 2008). Apart from that, there is some skepticism regarding the capability of NLP to support automated language learning. Salaberry (1996), for instance, argues that NLP cannot deal with the complexity of natural language. However, Nerbonne (2003) sur- mises that Salaberry’s skepticism is probably grounded in inflated expectations on the part of learners and teachers. Obviously, we are still a long way from perfectly imi- tating human-like language abilities in artificial systems. For instance, until today, no computer program has managed to pass the Turing test, i.e., make its behavior indis-

12 CHAPTER 2. COMPUTER-ASSISTED LANGUAGE LEARNING tinguishable from human behavior as judged by humans (Saygin et al., 2000; Shieber, 2004). Furthermore, despite the long history of machine translation (MT), current MT systems are still far from achieving the skills of a human translator. As Feigenbaum (2003) notes, for an artificial system to understand as well as a human is still an open challenge, despite the relative success in analyzing the syntactic structure of natural language. Gamper and Knapp (2002) summarize: “A full-fledged analysis of written text in all its complexity is a very difficult task, which exceeds current state of the art technology in NLP” (page 334).

However, while there are certainly some aspects of language complexity that are still hard to process, the usefulness of NLP in focused and controlled, if less ambitious, approaches has been convincingly demonstrated in a wide range of applications. For instance, the technology for morphological processing, that is, the analysis of the inner structure of words and how they are constructed of smaller meaningful units, known as morphemes, is sufficiently advanced. For many languages, it is mature and reliable enough to provide almost error-free lemmatization – deriving the canonical form of inflected word forms – as needed, for instance, in dictionary lookup tools (Nerbonne et al., 1998; Nerbonne, 2003).

Ambiguity

Contrary to that, syntactic and semantic analysis are much more challenging due to the inherent ambiguity of many sentences. There are two types of ambiguity, lexi- cal and structural. Lexical ambiguity refers to the fact that a word can have several meanings. Often, contextual information helps to disambiguate the word and arrive at the appropriate meaning. Structural ambiguity describes the fact that a sentence can have more than one possible syntactic structure, and consequently also more than one meaning. Consider as an example the sentence “I shot an elephant in my pajamas.” If its ambiguity is not apparent to the reader at first sight, it becomes evident when followed by the addendum how he got in my pajamas, I’ll never know1. The ambiguity is based on the prepositional phrase “in my pajamas”, which can specify either the ob- ject of the sentence (the elephant is wearing the pajamas) or the subject (the one who shoots is wearing pajamas).

Ambiguity and the analysis of learner errors

While ambiguity is already problematic within the domain of native and correct lan- guage, it is even more difficult for learner language, which is often incorrect. Erro- neous language is parsed with more difficulty, because the potential for ambiguity is increased. Amaral and Meurers (2011) explain that for the analysis of native and cor- rect language, the search space is constrained by lexical and syntactic rules. However, since learners are likely to violate these rules, the rules need to be extended to account for potentially ill-formed learner input. The expansion of rules increases the search space and thereby the number of possible ambiguities. Consider, for example, the learner production “The man eat cheese.”2 The sentence is incorrect according to stan-

1_{The joke is attributed to Groucho Marx in the film Animal Crackers.} 2_{Thanks to Detmar Meurers for this example.}

2.2. NATURAL LANGUAGE PROCESSING IN ICALL 13 dard English, but the source of the error and the intended meaning is unclear. The verb form eat cannot be used with third person singular subjects - so the learner might have used an incorrect verb form – it should be: The man eats cheese. But it is also possible that the learner intended to make a statement about several men and failed in producing the correct plural form – the correct sentence would be: The men eat cheese. Yet another possible source of an error in this sentence is the use of a wrong tense. If the learner wanted to express that the event has already taken place in the past, they might have failed in producing the past tense form ate. The correct sentence would be

The man ate cheese. In summary, there are at least three possible sources of errors for

this example.

Analyzing ill-formed language is hard because the deviations from correct input increase the space of possible analyses. These difficulties, however, have not deterred ICALL researchers and engineers from attempting to implement natural language pro- cessing facilities in their systems. Tokenizers, morphological analyzers, part-of-speech taggers, chunkers, tools for concordancing and text alignment, parsers, and semantic analysis tools have been successfully put to use (Amaral and Meurers, 2011; Nerbonne, 2003). We will describe some of these use cases in Section 3.2. However, developing tools for deeper analysis is a complex and costly endeavor (as noted, among others by Schulze (2008)). Developers have therefore sought for another approach to compen- sate for the increased difficulty of higher level analysis.

2.2.3 Constraining input as a strategy to deal with limits

A common strategy for dealing with the difficulties in analyzing learner language and for making processing tractable is to constrain the possible input that the learner can give to the system (Amaral and Meurers, 2011). The key is to do this in such a way that the learner does not feel too constrained and that the activity is still effective for fostering language skills. One very restrictive way to constrain the input is to let the learner choose from a set of pre-fabricated utterances, an approach taken, for instance, in the interactive systems described by Pollard and Yazdani (1993) or Stewart and File (2007). However, such restrictions eliminate the need for using NLP techniques alto- gether. A less constrained approach is taken for instance by Nagata (2009) or Heift (2003), who constrain learner input through the choice of task type, e.g., by prompting for a translation, dictating sentences, or providing a list of words that is to be used for the response. Another approach is to a priori constrain the input language to a sublanguage covered, for example, by a first-year textbook (Schwind, 1995; Levin and Evans, 1995). Schwind argues that a system should work on a sublanguage which is entirely known to the system, and the system “has to ensure that the student does only form sentences which can be analyzed, i.e., does not form well-formed sentences out- side the competence of the system” (page 296). Schwind further explains that “[t]his requirement is fulfilled by formulating the exercises so as to suggest a restricted lan- guage to the student”. Although she remains unspecific about how exactly to achieve this, she seems to imply the usage of more implicit ways of constraining the learner language. This is in accord with the desire to provide more freedom to the learner and “more space for negotiation of meaning as needed for meaning-based activities” (Amaral and Meurers, 2011, page 9). Amaral and Meurers propose to use pictures, lists

14 CHAPTER 2. COMPUTER-ASSISTED LANGUAGE LEARNING diagnosis approaches

language licensing

validity satisfiability

pattern matching

error patterns context patterns Figure 2.1– Taxonomy for error detection and diagnosis according to Meurers (2012)

of L2 words given as prompts or written cues in L2 to implicitly constrain the learner input to the system. Price et al. (1999) consider the task scenario of their dialog sys- tem (ordering meals in a restaurant) to be a sufficient restriction to the possible learner production.

The fact that adequate processing of largely unconstrained learner input is still beyond the current state of the art is confirmed by failures of projects that aimed at just that. Amaral and Meurers (2011) cite El Corrector (Klein, 1998) and FreeText (L’Haire, 2004) as examples of such disappointed expectations.

Constraining the learner input is related to the difference between emphasizing either meaning or formal correctness and explicit and implicit instruction. The more constrained the learner input is, the more likely this results in an explicit way of in- struction and one more focused on forms. The less constrained input is more likely to provide a more implicit instruction and a focus on meaning. We will discuss these pedagogic parameters in more detail in Chapter 4. For the study that we conduct in the scope of this thesis, we will come back to the issue of constraining learner input by using it as one of the implementational parameters that determine the developmental cost of an ICALL system.

After this general characterization of the state of the art in NLP for ICALL and its challenges, we will now discuss a particular relevant challenge for treating learner language.