Publication
Back-end semantics for multimodal dialog on XR devices
Peter Poller; Margarita Chikobava; Jack Hodges; Mareike Kritzler; Florian Michahelles; Tilman Becker
In: 26th International Conference on Intelligent User Interfaces-Companion. International Conference on Intelligent User Interfaces (IUI-21), April 13-17, Texas, USA, Association for Computing Machinery, 2021.
Abstract
Extended Reality (XR) devices have great potential to become the next wave in mobile interaction. They provide powerful, easy-touse Augmented Reality (AR) and/or Mixed Reality (MR) in conjunction with multimodal interaction facilities using gaze, gesture, and speech. However, current implementations typically lack a coherent semantic representation for the virtual elements, backendcommunication, and dialog capabilities. Existing devices are often restricted to mere command and control interactions. To improve these shortcomings and realize enhanced system capabilities and comprehensive interactivity, we have developed a flexible modular approach that integrates powerful back-end platforms using standard API interfaces. As a concrete example, we present our distributed implementation of a multimodal dialog system on the Microsoft Hololens®. It uses the SiAM-dp multimodal dialog platform as a back-end service and an Open Semantic Framework (OSF) back-end server to extract the semantic models for creating the dialog domain model.