Publikation
Interactive Multimodal Photobook Co-Creation in Virtual Reality
Sara-Jane Bittner; Robert Leist; László Kopácsi; Omair Shahzad Bhatti; Abdulrahman Mohamed Selim; Michael Barz; Daniel Sonntag (Hrsg.)
International Conference on Intelligent User Interfaces (IUI-2025), located at IUI-2025, March 24-27, Cagliari, Italy, ISBN 979-8-4007-1409-2, ACM, 2025.
Zusammenfassung
The integration of Multimodal-Multisensor Interface (MMI) technologies into Virtual Reality (VR) enables users to engage with computational systems in a natural and immersive way. However, these technologies remain underexplored when applied to deep learning (DL) systems in VR. This paper introduces a VR-based system designed to evaluate how users interact with DL models in virtual environments using MMI technologies, demonstrated through a photobook co-creation use case. The system facilitates human-AI collaboration (co-creation) by allowing users to work with DL models to create photobooks and supports incremental model learning based on user behaviour (Interactive DL) to produce personalised outputs. The tool features a Unity VR frontend that incorporates speech, gaze, and controller inputs. It has a modular backend architecture that allows seamless integration and testing of different DL models. This tool serves as a testbed for exploring MMI in immersive VR environments for both IDL and co-creation.