Publication
OpenMarcie: Dataset for Multimodal Action Recognition in Industrial Environments
Hymalai Bello; Lala Ray; Joanna Sorysz; Sungho Suh; Paul Lukowicz
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). International Conference on Computer Vision and Pattern Recognition (CVPR-2026), June 3-7, Denver, CO, USA, IEEE, 2026.
Abstract
Smart factories use advanced technologies to optimize production and increase efficiency. To this end, the recognition of worker activity allows for accurate quantification of
performance metrics, improving efficiency holistically while
contributing to worker safety. OpenMarcie is, to the best
of our knowledge, the biggest multimodal dataset designed
for human action monitoring in manufacturing environments. It includes data from wearables sensing modalities
and cameras distributed in the surroundings. The dataset
is structured around two experimental settings, involving a
total of 36 participants. In the first setting, twelve participants perform a bicycle assembly and disassembly task under semi-realistic conditions without a fixed protocol, promoting divergent and goal-oriented problem-solving. The
second experiment involves twenty-five volunteers (24 valid
data) engaged in a 3D printer assembly task, with the 3D
printer manufacturer’s instructions provided to guide the
volunteers in acquiring procedural knowledge. This setting
also includes sequential collaborative assembly, where participants assess and correct each other’s progress, reflecting
real-world manufacturing dynamics. OpenMarcie includes
over 37 hours of egocentric and exocentric, multimodal,
and multipositional data, featuring eight distinct data types
and more than 200 independent information channels. The
dataset is benchmarked across three human activity recognition tasks: activity classification, open vocabulary captioning, and cross-modal alignment. The dataset and code
are available at OpenMarcie.