Planteamiento y descripción del problema

Since the previous experiments date a few years back, we compared the performance of the in-domain SMT system from Section 6.2.3 with commercial systems using the latest MT technology, such as Google Translate and Systran. The results, as shown in Table 6.7, suggest that our system based on a moderate amount of in-domain texts is better than the commercial systems using considerably bigger amounts of general texts and possibly more recent MT technology, such as neural translation models. This finding is reflected by all the considered evaluation metrics, regardless if they are only based on exact matches (e.g. BLEU, TER) or also on word stems and paraphrases (e.g. METEOR). Moreover, this demonstrates that domain adaptation is still a powerful approach in Statistical MT and that the collected texts are valuable over time.

System BLEU ↑ METEOR ↑ TER ↓ In-domain (TB) 18.5 37.4 67.8

Google 16.0 35.0 71.5

Systran 13.3 33.5 75.3

Table 6.7: SMT results (DE-FR) for system configurations using Web-extracted data and in-domain data.

In order to gain a better understanding of these figures, we analyzed a few translations from the considered test set. We chose sentences with various translation performances on the same system, in order to avoid biased results. The examples reveal two main tendencies. As expected, the commercial SMT systems generally have difficulties in translating words from the Alpine domain, as it is clear from the first three examples. In the first one, the noun phrase eine steile, eher griffarme Rampe (EN: a steep ramp, with relatively few grips) poses difficulties: Google translates all words literally, but cannot convey the meaning of the adjective griffarm, whereas Systran even leaves it untranslated. In the second example, the commercial systems translate the noun Angriff with its most common equivalent in French attaque (EN: attack), whereas the desired translation would have been tentative (EN: attempt). In the third example, the word fl¨ussig is interpreted by the commercial system as an adjective and thus translated as liquide (EN: liquid), but in the original texts it was used as an adverb, as to express a continuous movement. In all these cases, our in-house MT system provided the correct translation in place.

Chapter 6. SMT Experiments 92 DE orig Es ist dies eine steile, eher griffarme Rampe.

EN gloss This is a steep ramp, with relatively few grips. FR ref C’est une rampe raide, plutˆot pauvre en prises. TB C’est une rampe raide, plutˆot pauvre en prises. Google Ceci est une plutˆot faible poign´ee rampe raide. Systran C’est cela une piste raide, plutˆot griffarme.

DE orig 20. Juni: unser dritter Angriff auf das Gross Gr¨unhorn (4044 m). EN gloss June 20th: our third attempt to the Gross Gr¨unhorn (4044 m). FR ref Le 20 juin eut lieu notre troisi`eme tentative au Gross Gr¨unhorn (4044

m).

TB Le 20 juin, notre troisi`eme tentative au Gross Gr¨unhorn (4044 m). Google 20 juin: notre troisi`eme attaque du Gr¨unhorn (4044 m).

Systran 20. juin: notre troisi`eme attaque sur grandement la corne d’ ´ecologiste ( 4044 m )

DE orig Wir klettern fl¨ussig weiter, schr¨ag aufw¨arts, der Wand entlang. EN gloss We climb up steadily along the mountain face.

FR ref Nous continuons `a grimper en nous ´elevant en diagonale le long de la paroi.

TB Nous grimpons vivement, en diagonale, le long de la paroi.

Google Nous montons encore liquide, obliquement vers le haut le long du mur. Systran Nous montons plus loin, en diagonale vers le haut, la paroi liquide. DE orig Die n¨achsten Montblanc-Anw¨arter sind bereits eingetroffen.

EN gloss The next candidates to the Mont Blanc have already arrived. FR ref Les prochains candidats au Mont Blanc sont d´ej`a arriv´es. TB Les prochaines Montblanc-Anw¨arter sont d´ej`a arriv´es. Google Montblanc pr´etendants `a venir sont d´ej`a arriv´es.

Systran Les prochains candidats de Mont-Blanc sont d´ej`a arriv´es.

DE orig Gegen Ende der Woche aber besserten sich die Verh¨altnisse zusehends. EN gloss By the end of the week the conditions improved significantly.

FR ref Mais, vers la fin de la semaine, les conditions s’am´elior`erent sensible- ment.

TB Vers la fin de la semaine, mais les conditions se besserten `a vue d’œil. Google Plus tard dans la semaine, mais les conditions am´elior´ees visiblement. Systran Vers la fin de la semaine, les relations se sont toutefois am´elior´ees

sensiblement.

Table 6.8: Example translations generated with our in-domain system and other commercial SMT systems.

The latter two examples illustrate the problem of untranslated words. In the fourth example, the German compound Montblanc-Anw¨arter was translated correctly only by the Systran system. Google successfully split it into parts (Montblanc pr´etendants),

Chapter 6. SMT Experiments 93 but translated them literaly and thus failed to generate a sound noun phrase, such as pr´etendants au Montblanc. Since our system trained on Alpine texts did not see the compound in the training phase, nor included a module for compound splitting, the compound was left untranslated. We think that our system would have been able to translate the individual words forming the compound if they would have been entered separately (Montblanc Anw¨arter ).

In the last example, the verb sich bessern (EN: to improve) could not be translated by our domain-specific system. Although the verb occured two times in the training corpus in the exactly same form, the automatic word alignments did not assign an equivalent for it. Google translated the main verb correctly, but omitted the reflexive pronoun and the auxiliary verb. Systran generated the translation most similar to the reference, though it included an inappropriate translation of the noun Verh¨altnisse (EN: conditions, relations), which is otherwise translated correctly by the other systems. A solution in this case would be to combine the available translation hypotheses in order to correct the shortcomings of the in-domain system, as suggested by Sennrich (2013). These examples provide further support of our idea that in-domain texts are extremely valuable when training SMT systems, as they represent the basis for learning the cor- rect translations in the context. This is of utmost importance for words with several meanings, one of which is frequent in general texts and another one occurring mostly in domain-specific texts. When enough in-domain data is used to train the SMT system, the translation probabilities will be automatically adjusted in order to give preference to domain-specific translations.

In Section 1.6 we showed that the vocabularies of an in-domain and an out-of-domain corpus vary a lot, in our case having an overlap of merely 26-35% (see Figure 1.6). This means that in-domain texts also contribute to the high lexical coverage of the SMT systems using them. This is also reflected by the higher evaluation scores achieved by the in-domain system. Although in some of the cases, the domain-specific system fails to translate words with little or no evidence in the underlying training corpus (e.g. compounds), its performance can be improved by using translations from other MT sytems.