Publication
EdTec-QBuilder: A Semantic Retrieval Tool for Assembling Vocational Training Exams in German Language
Alonso Palomino; Andreas Fischer; Jakub Kuzilek; Jarek Nitsch; Niels Pinkwart; Benjamin Paaßen
In: Kai-Wei Changi; Annie Lee; Nazneen Rajani (Hrsg.). Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: System Demonstrations. Annual Conference of the North American Chapter of the Association for Computational Linguistics (HLT-NAACL-2024), System Demonstrations, located at System Demonstrations Track, Mexico City, Mexico City, Mexico, ACL, 6/2024.
Abstract
Selecting and assembling test items from a validated item database into comprehensive exam forms is an under-researched but significant challenge in education. Search and retrieval methods provide a robust framework to assist educators when filtering and assembling relevant test items. In this work, we present EdTec-QBuilder, a semantic search tool developed to assist vocational educators in assembling exam forms. To implement EdTec-QBuilder's core search functionality, we evaluated eight retrieval strategies and twenty-five popular pre-trained sentence similarity models. Our evaluation revealed that employing cross-encoders to re-rank an initial list of relevant items is best for assisting vocational trainers in assembling examination forms. Beyond topic-based exam assembly, EdTec-QBuilder aims to provide a crowdsourcing infrastructure enabling manual exam assembly data collection, which is critical for future research and development in assisted and automatic exam assembly models.