Publication
Using a new analytic measure for the annotation and analysis of MT errors on real data
Arle Richard Lommel; Aljoscha Burchardt; Maja Popovic; Kim Harris; Eleftherios Avramidis; Hans Uszkoreit
In: Proceedings of the 17th Annual Conference of the European Association for Machine Translation. Annual Conference of the European Association for Machine Translation (EAMT-14), Proceedings of the 17th Annual Conference of the European Association for Machine Translation, June 16-18, Dubrovnik, Croatia, Pages 165-172, European Association for Machine Translation, 2014.
Abstract
This work presents the new flexible Multidimensional Quality Metrics (MQM) framework and uses it to analyze the performance of state-of-the-art machine translation systems, focusing on “nearly acceptable” translated sentences. A selection of WMT news data and “customer” data provided by language service providers (LSPs) in four language pairs was anno-
tated using MQM issue types and examined in terms of the types of errors found in it.
Despite criticisms of WMT data by the LSPs, an examination of the resulting errors and patterns for both types of data shows that they are strikingly consistent, with more variation between language pairs and system types than between text types. These results validate the use of WMT data in an analytic approach to assessing quality and show that analytic approaches represent a useful addition to more traditional assessment methodologies such as BLEU or METEOR.