Skip to main content Skip to main navigation

Publikation

Human-in-the-Loop Annotation for Image-Based Engagement Estimation: Assessing the Impact of Model Reliability on Annotation Accuracy

Sahana Yadnakudige Subramanya; Ko Watanabe; Andreas Dengel; Shoya Ishimaru (Hrsg.)
Human Computer Interaction International Conferences (HCII-2025), HCI INTERNATIONAL 2025, located at HCII-2025, June 22-27, Gothenburg, Sweden, Springer, 6/2025.

Zusammenfassung

Human-in-the-loop (HITL) frameworks are increasingly recognized for their potential to improve annotation accuracy in emotion estimation systems by combining machine predictions with human expertise. This study focuses on integrating a high-performing image-based emotion model into a HITL annotation framework to evaluate humanmachine interactions collaborative potential and uncover the psychological and practical factors critical to successful collaboration. Specifically, we investigate how varying model reliability and cognitive framing influence human trust, cognitive load, and annotation behavior in HITL systems. We show that model reliability and psychological framing significantly impact annotators trust, engagement, and consistency, offering insights into optimizing HITL frameworks. Through three experimental scenarios with 29 participants baseline model reliability (S1), fabricated errors (S2), and cognitive bias introduced by negative framing we analyzed behavioral and qualitative data (S3). Reliable predictions (S1) yielded high trust and annotation consistency, while unreliable outputs (S2) induced critical evaluations but increased frustration and response variability. Negative framing (S3) revealed how cognitive bias influenced participants to rate the model as relatable and accurate despite misinformation about its reliability. These findings highlight the importance of reliable machine outputs and psychological factors in shaping effective human-machine collaboration. By leveraging the strengths of both human oversight and automated systems, this study establishes a scalable HITL framework for emotion annotation and sets the stage for broader applications in adaptive learning and human-computer interaction.

Projekte

Weitere Links

https://arxiv.org/abs/2502.07404
_HCII__Human_in_the_Loop_Annotation_for_Image_Based_Engagement_Estimation__Assessing_the_Impact_of_Model_Reliability_on_Annotation_Accuracy.pdf (pdf, 945 KB )