Publikation
Overview of the CLEF 2008 Multilingual Question Answering Track
Pamela Forner; Anselmo Peñas; Iñaki Alegria; Corina Forascu; Nicolas Moreau; Petya Osenova; Prokopis Prokopidis; Paulo Rocha; Bogdan Sacaleanu; Richard Sutcliffe; Erik Tjong Kim Sang
In: Carol Peters et al. (Hrsg.). Working Notes for the CLEF 2008 Workshop. Cross Language Evaluation Forum (CLEF-08), September 17-19, Aarhus, Denmark, Springer, 2008.
Zusammenfassung
The QA campaign at CLEF [1], was manly the same as that proposed last year. The results and the analyses reported by last year's participants suggested that the changes introduced in the previous campaign had led to a drop in systems' performance. So for this year's competition it has been decided to practically replicate last year's exercise.
Following last year's experience some QA pairs were grouped in clusters. Every
cluster was characterized by a topic (not given to participants). The questions from
a cluster contained co-references between one of them and the others. Moreover,
as last year, the systems were given the possibility to search for answers in Wikipedia1
as document corpus beside the usual newswire collection.
In addition to the main task, three additional exercises were offered, namely the
Answer Validation Exercise (AVE), the Question Answering on Speech Transcriptions
(QAST), which continued last year's successful pilot, and Word Sense Disambiguation for Question Answering (QA-WSD).
As general remark, it must be said that the task still proved to be very challenging
for participating systems. In comparison with last year's results the Best Overall
Accuracy dropped significantly from 41,75% to 19% in the multi-lingual subtasks, while instead it increased a little in the monolingual sub-tasks, going from 54% to 63,5%.