Publikation
Task-oriented Dependency Parsing Evaluation Methodology
Alexander Volokh; Günter Neumann
In: IEEE 13th International Conference on Information Reuse and Integration. IEEE International Conference on Information Reuse and Integration (IRI-2012), 13th, August 8-10, Las Vegas, NV, USA, IEEE Systems, Man, and Cybernetics Society (SMC), 2012.
Zusammenfassung
Traditional parser evaluation with attachment scores is not helpful for researchers who want to find the most suitable parser for their application. First, because it is being done for a domain which is almost always different from the domain of the application and second because many of the tested dependencies are irrelevant for the application. The alternative extrinsic evaluation is problematic as well, since it is difficult to find a suitable data set and because it is not straightforward how to measure the quality of the parser in the context of a broader appllication. We propose a method which combines the strengths of attachment scores and extrinsic evaluation and avoids their weaknesses. On the one hand we use the very robust attachment scores, We apply our approach to RTE-7 data in order to demonstrate how it works.