Skip to main content Skip to main navigation


Linguistically Motivated Evaluation of the 2023 State-of-the-art Machine Translation: Can GPT-4 Outperform NMT?

Shushen Manakhimova; Eleftherios Avramidis; Vivien Macketanz; Ekaterina Lapshinova-Koltunski; Sergei Bagdasarov; Sebastian Möller
In: Proceedings of the Eighth Conference on Machine Translation (WMT). Conference on Machine Translation (WMT-2023), located at The 2023 Conference on Empirical Methods in Natural Language Processing, December 6-7, Singapore, Singapore, Association for Computational Linguistics, 12/2023.


This paper offers a fine-grained analysis of the machine translation outputs in the context of the Shared Task at the 8th Conference of Machine Translation (WMT23). Building on the foundation of previous test suite efforts, our analysis includes Large Language Models and an updated test set featuring new linguistic phenomena. To our knowledge, this is the first fine-grained linguistic analysis for the GPT-4 translation outputs. Our evaluation spans German--English, English--German, and English--Russian language directions. Some of the phenomena with the lowest accuracies for German--English are idioms and resultative predicates. For English--German, these include mediopassive voice, and noun formation(er). As for English--Russian, these included idioms and semantic roles. GPT-4 performs equally or comparably to the best systems in German--English and English-–German, but falls in the second significance cluster for English–Russian.