How Is Machine Translation Quality Measured?

How do you decide whether a translation engine is any good? In most organisations that decision rests on a trial of a few sentences and a general impression. Yet machine translation quality measurement is a field with a wide literature and well established methods behind it.

Contents

The hard part of measurement is this: translation has no single right answer. The same sentence can have more than one valid rendering. Measurement is therefore less like confirming one option among several and more like assessing a range.

In this article we take up three families of measurement, what each of them is good for, and a test design you can apply to your own documents.

Why is machine translation quality measurement difficult?

The first difficulty of measurement is the problem of multiplicity. A sentence has a great many valid translations, and which of them is better depends on the purpose of the text and on its readership. The freedom that counts as good in a promotional text is a fault in a contract.

The second difficulty is that quality is not one-dimensional. An output can be fluent but wrong, or accurate but foreign to the ear. Unless these two dimensions are measured separately, a single verdict of “good” or “bad” carries no information.

The third difficulty is the problem of the sample. A trial of five sentences says nothing about how the engine behaves on long sentences, on domain terminology or on formatting elements. When the sample is small and hand-picked, the result is an impression, not a measurement.

The fourth difficulty is variability. The same engine behaves differently on different document types. The sentence “this engine is good” is incomplete; the correct sentence takes the form “this engine is good on this document type”.

Human evaluation

The oldest method, and still the most reliable, is human evaluation. The classic design scores two dimensions separately: adequacy and fluency.

Adequacy asks how much of the information in the source text has been carried across into the target text. To be able to measure this dimension, the evaluator has to see the source text. Fluency asks how natural the target text is in its own right; for that there is no need to look at the source.

Separating the two is critical. As neural models spread, fluency scores rose markedly; fluency, however, is not proof of adequacy. An evaluation carried out by having only the output read gives systematically optimistic results.

The price of human evaluation is cost, and consistency. Different evaluators can give the same text different scores. The scoring scale and the instructions should therefore be written down in advance, and more than one evaluator used where possible.

Automatic metrics

Automatic metrics were developed to get past the cost of human evaluation. Their shared logic is this: compare the machine output with a reference translation made by a human and put a number on the similarity.

Metric	What does it look at?	Where it is strong
BLEU (Papineni et al. 2002)	Overlap of word sequences with the reference	Fast comparison between systems
TER (Snover et al. 2006)	The number of edits needed to reach the reference	Estimating the editing burden
chrF	Overlap at character level	Affix-rich languages
COMET (Rei et al. 2020)	Closeness in meaning, based on a neural model	Higher agreement with human judgement

The advantage these metrics share is that they are cheap and repeatable. Whether a change made to an engine has improved the output can be measured within minutes.

The reason metrics that work at character level are recommended for affix-rich languages is structural. A metric that looks at word level counts two forms differing only in their affixes as entirely different and gives an unfairly low score.

The limits of automatic metrics

The most basic limit is dependence on the reference. A metric measures not whether the output is correct but how closely it resembles a single reference. A translation that differs from the reference yet is perfectly valid receives a low score.

The second limit is that errors are not weighted. A shift in a word sequence and the loss of a negation carry similar weight in most metrics. Yet the second reverses the meaning of the text, while the first is only a matter of style.

The third limit is comparability. Scores obtained on different texts and on different language pairs cannot be compared with one another. The value the same metric returns on one language pair does not mean the same thing as the value on another.

Automatic metrics should therefore be used for relative comparison, not for an absolute verdict on quality. Comparing two engines, or two versions of one engine, on the same text is reasonable; saying “that score is good enough” is not.

Evaluation based on error typology

The third family, instead of reducing quality to a single score, sorts errors into types and counts them. The most widespread framework for this approach in the industry is the family of models built on an error typology; every error is placed in a category and given a weight.

The typical categories are these: accuracy errors (omission, addition, mistranslation), fluency errors (grammar, spelling, punctuation), terminology errors, formatting errors and locale convention errors (date, number and currency formats).

The strength of this approach is that the result can be acted on. Instead of saying “the score is low” you can say “terminology errors are heavy”, and that points straight to a solution, setting up a term list for instance.

The price is labour; the classification is done by hand. It can be carried out even on a small sample, however, and the result is far more informative than an automatic score taken from that same sample.

The reference translation: the hidden cost of measurement

Every automatic metric needs a reference translation. That reference has to have been prepared by a human, independently of the text being measured. In other words, before you measure anything you must already hold a human translation of that text.

This is a cost most organisations fail to allow for. Preparing a reference means having as much translated as the sample you intend to measure. The larger the sample, the higher the cost; the smaller it is, the less reliable the measurement.

The good news is that most organisations already hold a usable pool of references. Documents translated by hand and approved in earlier years are exactly right for this purpose. Here too, keeping the archive in order pays off directly.

One point needs attention when choosing the reference: the reference is itself an interpretation. Taking a freely translated, adapted or abridged text as the reference ruins the measurement from the start. References should be chosen from translations that stay as close to the source as possible.

Who should measure, and how often?

Who carries out the measurement has a direct bearing on the validity of the result. The person doing the evaluation needs both a good command of the target language and a firm grasp of the subject of the text. An evaluator without knowledge of the field cannot spot terminology errors.

There is no single rule on frequency, but measurement is not a one-off exercise. Engines are updated, your document types change and your supplier may alter its processes. A small check once a year catches drifts in quality early.

Keeping the results of the measurement matters too. Records that build up over time form a body of evidence you can use in supplier negotiations and in internal reporting. This is what takes the quality discussion off impression and onto data.

One last caution: measurement is itself a cost and should not outgrow its purpose. For an organisation that has a few thousand words translated a year, setting up a detailed evaluation framework makes no sense. The depth of the measurement should be proportionate to your translation volume and to your exposure to error.

There is one limit to keep in mind when reading measurement results: these indicators measure the fluency of the text and its closeness to the reference, not the fitness of the document for its purpose. Whether it will be accepted in an official application is not a question any automatic metric is able to answer.

The Türkiye context

For Turkish, quality measurement has to be set up more carefully than international practice would suggest, for several reasons. The first is the choice of metric: metrics that work at word level tend to produce systematically low scores because of the affixal structure of Turkish. Metrics that work at character level give fairer results in this respect.

The second is the problem of the reference. Because constituent order is free in Turkish, the same sentence has a great many valid orderings. Measurement based on a single reference can count this variety as a fault. Automatic scores in Turkish should therefore be read together with human evaluation.

The third is the problem of priority. In Turkish, most of the errors that reverse meaning hide in small suffixes: the negation suffix, the person suffix, the question particle. The first items on a Turkish evaluation list should therefore be these suffixes rather than style. In official documents, date, currency and identity number formats are a separate checking item.

What does this mean for you?

Quality measurement does not have to be a large project. A single day’s work can give your organisation a meaningful baseline. The suggested order is as follows:

1. Pick five documents at random from your archive; take the typical ones, not the easiest ones.
2. Have the documents translated with no correction at all and set the output side by side with the source.
3. Have an employee who knows the subject read it, and have them enter every error they find into one of five categories: omission, addition, terminology, number and format, grammar.
4. Total up the categories; whichever category dominates is where the solution lies.
5. Repeat the same test six months later with the same documents and compare the results.

This table makes every conversation concrete, from choosing a supplier to negotiating a price. If you would like us to design the evaluation around your own documents, you can get in touch, and for our other articles on similar subjects you can look through our knowledge centre.

Frequently asked questions

How is machine translation quality measured?

There are three main routes: human evaluation, automatic metrics and scoring based on error typology. The three answer different questions; a serious evaluation usually uses at least two of them together.

What do automatic metrics measure?

They measure how closely the machine output resembles a reference translation made by a human. Similarity is not the same as accuracy; a translation that differs yet is valid can score low.

How do I run a test with my own text?

Pick three to five documents at random from your archive, have them translated with no correction and have an employee who knows the subject read them against the source. Sort the errors found by type and count them; the decision is taken on that table.

Which error types should count as the most critical?

Errors that reverse meaning come first: loss of a negation, an error of person, an error in a number or a unit, information added that is not in the source. Flaws of style usually come second.

References

• Papineni, K. et al. (2002). BLEU: a Method for Automatic Evaluation of Machine Translation. ACL 2002, pp. 311–318.
• Snover, M. et al. (2006). A Study of Translation Edit Rate with Targeted Human Annotation. AMTA 2006, pp. 223–231.
• Rei, R. et al. (2020). COMET: A Neural Framework for MT Evaluation. EMNLP 2020, pp. 2685–2702.
• Pietrzak, P. & Kornacki, M. (2021). Using CAT Tools in Freelance Translation. Routledge Focus, Chapter 3.