DFKI and Inria Develop Robust AI for Sign Language Translation

Existing sign language systems can already convert spoken or written language into sign language and visualize it with avatars. In practice, however, they reach their limits: facial expressions and body language are often rendered inaccurately, aspects of meaning are lost, and the representation frequently appears unnatural.

This three-year project addresses these issues: It improves both the translation of text into sign language and the conversion of sign-language videos into written language. Modern methods such as multimodal neural networks, self-supervised learning, and large language models are intended to overcome existing hurdles—such as limited data availability, lack of generalizability, and unnatural translations. A key component is the development of new data resources, including extensive parallel corpora of sign-language videos and associated texts.
 

The project will culminate in a prototype for bidirectional automatic translation – between German and DGS as well as French and LSF. The signed output is delivered via an avatar that realistically portrays the content through full-body movements, facial expressions, and emotional expression.

In addition to scientific advances, RoGSiLT pursues a clear societal goal: The technologies developed are intended to break down barriers and open new communication opportunities—in education, the workplace, and access to information. In this way, the project contributes to the implementation of European and international strategies for inclusion and accessibility, such as the European Accessibility Act and the UN Convention on the Rights of Persons with Disabilities.
 

RoGSiLT is already the ninth joint project between DFKI and Inria. The collaboration builds on existing strengths: Both partners contribute extensive datasets—including sign-language data from previous research projects and more than 300 hours of video footage from news broadcasts. At the same time, their areas of expertise complement each other: While Inria has special expertise in machine learning and spoken language, DFKI has many years of experience modeling sign-language elements.