Publication

Interactive Machine Learning for Image Captioning

Mareike Hartmann, Aliki Anagnostopoulou, Daniel Sonntag

The AAAI-22 Workshop on Interactive Machine Learning 2/2022.

Abstract

We propose an approach for interactive learning for an image captioning model. As human feedback is expensive and modern neural network based approaches often require large amounts of supervised data to be trained, we envision a system that exploits human feedback as good as possible by multiplying the feedback using data augmentation methods, and integrating the resulting training examples into the model in a smart way. This approach has three key components, for which we need to find suitable practical implementations: feedback collection, data augmentation, and model update. We outline our idea and review different possibilities to address these tasks.

Projekte

interactive_learning_for_image_captioning.pdf (pdf, 964 KB )

German Research Center for Artificial Intelligence
Deutsches Forschungszentrum für Künstliche Intelligenz