Multilinguality and Language Technology

Modern NLP models and LLMs have specific flaws, despite being highly performant: First, they are black boxes: Parameters of proprietary models are not accessible at all; and even non-proprietary models are largely opaque in the sense that it is unclear where exactly specific knowledge is encoded in potentially billions of parameters. Second, there is a tendency to always increase the size of LLMs and training data to improve performance, which is especially problematic for domains or languages with fewer resources.

The E&E group of DFKI’s Research Department Multilinguality and Language Technology works on transparent and efficient NLP models. Our objective is to make the parameters and behaviour of LLMs more explainable and understandable to both end users and researchers. We try to improve LLMs with regard to data consumption, e.g. for domains or languages where data is scarce, by using structured data, new learning techniques, or other modalities; and in terms of model size, e.g. for settings where powerful hardware is not available.

We are involved in Twinning projects, where we provide knowledge transfer both on research topics and project management to newly established research institutions across Europe. We are involved in European procurement projects focusing on language resources, such as the European Language Resource Coordination and the Language Data Space.

• Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization
  Cennet Oguz; Pascal Denis; Emmanuel Vincent; Simon Ostermann; Josef van Genabith
  In: Houda Bouamor; Juan Pino; Kalika Bali (Hrsg.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing (EMNLP), Singapore, Pages 8099-8110, Association for Computational Linguistics, 2023.
• Investigating the Encoding of Words in BERT's Neurons Using Feature Textualization
  Tanja Bäumel; Soniya Vijayakumar; Josef van Genabith; Günter Neumann; Simon Ostermann
  In: Yonatan Belinkov; Sophie Hao; Jaap Jumelet; Najoung Kim; Arya McCarthy; Hosein Mohebbi (Hrsg.). Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP. Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP-2023), Singapore, Pages 261-270, Association for Computational Linguistics, 2023.
• InterroLang: Exploring NLP Models and Datasets through Dialogue-based Explanations
  Nils Feldhus; Qianli Wang; Tatiana Anikina; Sahil Chopra; Cennet Oguz; Sebastian Möller
  In: Houda Bouamor; Juan Pino; Kalika Bali (Hrsg.). Findings of the Association for Computational Linguistics: EMNLP 2023. Conference on Empirical Methods in Natural Language Processing (EMNLP-2023), December 6-10, Singapore, Singapore, Association for Computational Linguistics, 12/2023.
• HybridBERT - Making BERT Pretraining More Efficient Through Hybrid Mixture of Attention Mechanisms
  Gokul Srinivasagan and Simon Ostermann
  In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 4: Student Research Workshop), pages 285–291, Mexico City, Mexico. Association for Computational Linguistics. Runner-Up Best Paper Award.
• MMAR: Multilingual and Multimodal Anaphora Resolution in Instructional Videos
  Cennet Oguz, Pascal Denis, Simon Ostermann, Emmanuel Vincent, Natalia Skachkova, and Josef van Genabith
  In Findings of the Association for Computational Linguistics: EMNLP 2024. Association for Computational Linguistics.
• Common European Language Data Space
  Georg Rehm, Stelios Piperidis, Khalid Choukri, Andrejs Vasiļjevs, Katrin Marheinecke, Victoria Arranz, Aivars Bērziņš, Miltos Deligiannis, Dimitris Galanis, Maria Giagkou, Katerina Gkirtzou, Dimitris Gkoumas, Annika Grützner-Zahn, Athanasia Kolovou, Penny Labropoulou, Andis Lagzdiņš, Elena Leitner, Valérie Mapelli, Hélène Mazo, et al.. 2024.
  In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 3579–3586, Torino, Italia. ELRA and ICCL.