Publication
Factoid and Open-Ended Question Answering with BERT in the Museum Domain
Md Mahmud Uz Zaman; Stefan Schaffer; Tatjana Scheffler
In: Proceedings of the Conference on Digital Curation Technologies. Conference on Digital Curation Technologies (QURATOR-2021), February 8-12, Berlin/Virtual, Germany, CEUR Workshop Proceedings, 2021.
Abstract
Most question answering tasks are oriented towards open domain factoid questions. In comparison, much less work has studied both factoid and open ended questions in closed domains. We have chosen a current state-of-art BERT model for our question answering experiment, and investigate the e ectiveness of the BERT model for both factoid and open-ended questions in the museum domain, in a realistic setting. We conducted a web based experiment where we collected 285 questions relating to museum pictures. We manually determined the answers from the description texts of the pictures and classi ed them into answerable/un-answerable and factoid/open-ended. We passed the questions through a BERT model and evaluated their performance with our created dataset. Matching our expectations, BERT performed better for factoid questions, while it was only able to answer 36% of the open-ended questions. Further analysis showed that questions that can be answered from a single sentence or two are easier for the BERT model. We have also found that the individual picture and description text have some implications for the performance of the BERT model. Finally, we propose how to overcome the current limitations of out of the box question answering solutions in realistic settings and point out important factors for designing the context for getting a better question answering model using BERT.