Машиналық аударма мәтінінің сапасын бағалау алгоритмі

Algorithm for evaluating the quality of machine translated text
the evaluation metric for machine translation
NOTOC
BLEU (bilingual evaluation understudy) is an algorithm for evaluating the quality of text which has been machine translated from one natural language to another. Quality is considered to be the correspondence between a machine's output and that of a human: "the closer a machine translation is to a professional human translation, the better it is" – this is the central idea behind BLEU. Invented at IBM in 2001, BLEU was one of the first metrics to claim a high correlation with human judgements of quality, and remains one of the most popular automated and inexpensive metrics. Scores are calculated for individual translated segments—generally sentences—by comparing them with a set of good quality reference translations. Those scores are then averaged over the whole corpus to reach an estimate of the translation's overall quality. Intelligibility or grammatical correctness are not taken into account. BLEU's output is always a number between 0 and 1. This value indicates how similar the candidate text is to the reference texts, with values closer to 1 representing more similar texts. Few human translations will attain a score of 1, since this would indicate that the candidate is identical to one of the reference translations. For this reason, it is not necessary to attain a score of 1. Because there are more opportunities to match, adding additional reference translations will increase the BLEU score.

A basic, first attempt at defining the BLEU score would take two arguments: a candidate string and a list of reference strings The idea is that should be close to 1 when is similar to , and close to 0 if not. As an analogy, the BLEU score is like a language teacher trying to score the quality of a student translation by checking how closely it follows the reference answers
Since in natural language processing, one should evaluate a large set of candidate strings, one must generalize the BLEU score to the case where one has a list of M candidate strings (called a "corpus") , and for each candidate string , a list of reference candidate strings
Given any string , and any integer , we define the set of its n grams to be Note that it is a set of unique elements, not a multiset allowing redundant elements, so that, for example,

Given any two strings , define the substring count to be the number of appearances of as a substring of For example,
Now, fix a candidate corpus , and reference candidate corpus , where each .

Define the modified n gram precision function to be The modified n gram, which looks complicated, is merely a straightforward generalization of the prototypical case: one candidate sentence and one reference sentence. In this case, it is To work up to this expression, we start with the most obvious n gram count summation:
This quantity measures how many n grams in the reference sentence are reproduced by the candidate sentence. Note that we count the n substrings, not n grams. For example, when , all the 2 substrings in (ab and ba) appear in 3 times each, so the count is 6, not 2. In the above situation, however, the candidate string is too short. Instead of 3 appearances of it contains only one, so we add a minimum function to correct for that: This count summation cannot be used to compare between sentences, since it is not normalized. If both the reference and the candidate sentences are long, the count could be big, even if the candidate is of very poor quality. So we normalize it The normalization is such that it is always a number in , allowing meaningful comparisons between corpuses. It is zero iff none of the n substrings in candidate is in reference. It is one iff every n gram in the candidate appears in reference, for at least as many times as in candidate. In particular, if the candidate is a substring of the reference, then it is one.

There is not a single definition of BLEU, but a whole family of them, parametrized by the weighting vector It is a probability distribution on , that is, , and
With a choice of , the BLEU score isIn words, it is a weighted geometric mean of all the modified n gram precisions, multiplied by the brevity penalty. We use the weighted geometric mean, rather than the weighted arithmetic mean, to strongly favor candidate corpuses that are simultaneously good according to multiple n gram precisions. The most typical choice, the one recommended in the original paper, is .

BLEU has frequently been reported as correlating well with human judgement, and remains a benchmark for the assessment of any new evaluation metric. There are however a number of criticisms that have been voiced. It has been noted that, although in principle capable of evaluating translations of any language, BLEU cannot, in its present form, deal with languages lacking word boundaries. Designed to be used for several reference translation, in practice it's used with only the single one. BLEU is infamously dependent on the tokenization technique, and scores achieved with different ones are incomparable (which is often overlooked); in order to improve reproducibility and comparability, SacreBLEU variant was designed. It has been argued that although BLEU has significant advantages, there is no guarantee that an increase in BLEU score is an indicator of improved translation quality.