A parallel corpus consists of a collection of texts that have been translated into one or more other languages. Over the past 10 to 15 years, the increased availability of computational power, memory, storage, and parallel texts has enabled active research in the field of corpus-based translation — using manually-translated texts as the basis for translating new texts. Corpus-based machine translation can be subdivided into three categories: translation memories, example-based machine translation, and statistical machine translation.
Translation memories simply store prior reference translations of text and retrieve the nearest match in their database. It is then up to the human translator to clean up the retrieved translation and to account for the differences between the entry in the database and the actual passage to be
translated. This very simple technology has resulted in a major increase in human translator pro- ductivity, and is commercially available in several products, including IBM’s Translation Manager (TM2), STAR’s Transit, and SDL International’s SDLX.
Example-based machine translation approaches are often characterized by their use of a bilin- gual corpus as their main knowledge base, at run-time. It is essentially a translation by analogy and can be viewed as an implementation of case-based reasoning [Brown, 1996]. This method is grounded on the conviction that there are no pre-established solutions to translation, but most possible solutions can be found in texts already translated by professionals. In other words, a large portion of a translator’s competence is encoded in the language equivalences that can be found in already translated texts.
Statistical machine translation systems use a parallel training corpus to build a probabilistic model of the possible translations for words and the reordering of words between languages. After training, the corpus is no longer required, as all translation is performed using the statistical model. Approaches that involve training a statistical translation model have been explored by, for example, Nie et al. [1999] and Franz et al. [2001]. In Nie et al.’s approach, statistical translation models (usually IBM model 1) are trained on a parallel corpus. The models are used in a straightforward way: the source query is submitted to the translation model, which proposes a set of translation equivalents, together with their probabilities. The latter are then used as a query for the retrieval process, which is based on a vector space model. Franz et al.’s approach uses a better founded theoretical framework: the OKAPI probabilistic IR model. The present study uses a different probabilistic IR model, one based on statistical language models [Hiemstra et al., 2001; Xu et al., 2001]. This IR model facilitates a tighter integration of translation and retrieval. An important difference between statistical translation approaches and approaches based on document alignment discussed above is that translation models use alignment at a much more refined level. Consequently, the alignments can be used to estimate translation relations in a reliable way. On the other hand, the advantage of the CLIR approaches that rely only on alignment at the document level is that they can also handle comparable corpora, that is, documents that discuss the same topic but are not necessarily translations of each other.
Most previous work on parallel texts has used a few manually constructed parallel corpora, notably the Canadian Hansard corpus. This corpus contains many years’ debates in the Canadian parliament in both English and French, amounting to several dozens of millions of words in each language. Documents from the European parliament represent another large parallel corpus in several European languages. However, the availability of this corpus is much more restricted than
the Canadian Hansard. The Hong Kong government publishes official documents in both Chinese and English. They form a Chinese–English parallel corpus, but again, its size is much smaller than that of the Canadian Hansard. For many other languages, no large parallel corpora are available for the training of statistical models. The explosion of information on the web yields an interesting new source of parallel text, which researchers have been harnessing through automated retrieval from the web. Finding a realistic, systematic, and cost-effective way for automated acquisition of bilingual parallel corpora remains an open challenge in corpus-based CLIR.
Kraaij et al. [2003] developed the PTMiner to mine large parallel corpora from the web. PT- Miner used search engines to pinpoint candidate sites that are likely to contain parallel pages, and then used the URLs collected as seeds to further crawl each web site for more URLs. The pairs of web pages were extracted on the basis of manually defined URL pattern-matching, and further filtered according to criteria such as file length, HTML structure, and language character set. Resnik and Smith [2003] at the University of Maryland reported a similar effort. Their approach is to use a web spider to collect pages containing certain key expressions that tend to indicate that a certain link points at a translated version of the page, and then filter the retrieved pairs of pages by ensuring structural parallelism of the HTML tags within the pages. However, it is usually not only time consuming but also expensive to acquire large high-quality parallel bilingual corpora, particularly for minor languages.