Statistical Machine Translation

1. Introduction

Statistical Machine Translation (SMT) is an approach to automatically translate text based on the use of statistical models and examples of translations. Although Machine Translation (MT) systems developed according to rule-based approaches are still in use, SMT is the dominant research paradigm today and has recently been garnering significant commercial interest. The core of SMT research has developed over the last two decades, after the seminal paper by Brown et al. (1990). The field has progressed considerably since then, moving from word-to-word translation towards phrase-to-phrase translation and other more sophisticated models that take sentence structure into account. A trend observed in recent years is the shift from using pure statistical information extracted from large quantities of data to incorporating linguistic information about the source and/or the target language.

The idea of SMT is related to the late 1940’s view of the translation task as a cryptography problem where a decoding process is needed to translate from a foreign “code” into the English language (Hutchins, 1997). This is the basis for the fundamental approach to SMT proposed in the early 1990s through the application of the Noisy Channel Model (Shannon, 1949) from the field of Information Theory. This model had proved to be successful in the area of Speech Recognition and was thus adapted to MT.

The use of the Noisy Channel Model for translation assumes that the original text has been accidentally scrambled or encrypted (using a different alphabet, for example) and the goal is to find out the original text by “decoding” the encrypted/scrambled version, as depicted in Figure 1. According to this model, the message I is the input to the channel (text in a native language). I gets encrypted into O (text in a foreign language) using a certain coding scheme. The goal is to find a decoder that can reconstruct the input message as faithfully as possible into I*.

Figure 1.

The noisy channel model

In a probabilistic framework, finding I*, i.e., the closest possible text to I, can be stated as finding the argument that maximizes the probability of recovering the original input given the noisy text, i.e.:

This problem is commonly exemplified as the task of translating from a foreign language sentence f into an English sentence e. Given f, we seek the translation e that maximizes P(e|f), i.e., the most likely translation:

This problem can be decomposed in smaller and simpler problems applying the Bayes Theorem:

Since the source text f, i.e., the input for the translation task is constant across all possible translations e, P(f) can be disregarded:

The process of decomposing the problem of translation into smaller problems and modeling each step individually is motivated by the fact that more reliable statistics can be collected for the smaller problems. The modeling of each smaller problem with probability distributions followed by their combination to find a model that best explains the data complies with a type of statistical learning called generative modeling. The generative models resulting from the decomposition of P(e|f) correspond to two of the fundamental components of a basic SMT system: the translation modelP(f|e) and the language modelP(e). The translation model is used to search for the best translation given an input text. While mathematically it represents the inverse translation probability, i.e., the probability of the source text given the target text, in practice the initial direction, i.e., the probability of the target text given the source text, can be estimated in the very same way using the same data, as we will discuss later. In most SMT systems, both probability directions are used. The decomposition is relevant mostly to isolate the language model component from the translation model. The language model searches for the best translation regardless of the input text.