Publikation
Find-2-Find: Multitask Learning for Anaphora Resolution and Object Localization
Cennet Oguz; Pascal Denis; Emmanuel Vincent; Simon Ostermann; Josef van Genabith
In: Houda Bouamor; Juan Pino; Kalika Bali (Hrsg.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing (EMNLP), Singapore, Pages 8099-8110, Association for Computational Linguistics, 2023.
Zusammenfassung
In multimodal understanding tasks, visual and linguistic ambiguities can arise. Visual ambiguity can occur when visual objects require a model to ground a referring expression in a video without strong supervision, while linguistic ambiguity can occur from changes in entities in action flows. As an example from the cooking domain, ``oil'' mixed with ``salt'' and ``pepper'' could later be referred to as a ``mixture''. Without a clear visual-linguistic alignment, we cannot know which among several objects shown is referred to by the language expression ``mixture'', and without resolved antecedents, we cannot pinpoint what the mixture is. We define this chicken-and-egg problem as Visual-linguistic Ambiguity. In this paper, we present Find2Find, a joint anaphora resolution and object localization dataset targeting the problem of textitvisual-linguistic ambiguity, consisting of 500 anaphora-annotated recipes with corresponding videos. We present experimental results of a novel end-to-end joint multitask learning framework for Find2Find that fuses visual and textual information and shows improvements both for anaphora resolution and object localization with one joint model in multitask learning, as compared to a strong single-task baseline.