Publication
Automatic Alignment between Sign Language Videos and Motion Capture Data: A Motion Energy-Based Approach
Fabrizio Nunnari; Mina Ameli; Shailesh Mishra
In: Proceedings of the Eighth International Workshop on Sign Language Translation and Avatar Technology. International Workshop on Sign Language Translation and Avatar Technology (SLTAT-2023), located at ICASSP 2023, June 10, Rhodes, Greece, IEEE, 2023.
Abstract
In this paper, we propose a method for the automatic alignment of sign language videos and their corresponding motion capture data, useful for the preparation of multi-modal sign language corpora. First, we extract an estimate of the motion energy from both the video and the motion capture data. Second, we align the two curves to minimize their distance. Our tests show that it is possible to achieve a mean absolute error as low as 1.11 frames using optical flow for video energy extraction and a set of 22 bones for skeletal energy extraction.