There is a growing awareness within the natural language (NL) community of the need for adequate assessment and evaluation methods and techniques for NL systems. One technique is the use of the test suite as a diagnostic evaluation tool. The problem lies in defining an appropriate test suite for a given system.
The TSNLP project aims at a considerable improvement in the informativeness and functionality of test suites by performing a thorough study of design issues, devising an efficient annotation scheme, automating the construction of test data, and designing and implementing a database for the storage and manipulation of test data. The project will produce a set of guidelines for the construction of test suites for a range of NL products. Important issues in the design of test suites will be identified and addressed for each application type. In addition, the project aims at producing test data in three languages: English, French and German.
TSNLP will also develop a number of tools which will facilitate the construction and use of test suites, such as a tool which allows the automatic generation of test data, as well as a database will be developed in which the test data will be stored. This allows easy access and manipulation of the data, e.g. the retrieval of those test sentences or sentence fragments required for the testing of a particular phenomenon.
It is envisaged that by achieving the proposed improvements on test suite design, construction and manipulation the use of diagnostic test data for system evaluation will become more informative and efficient. By making the results of the project public domain, it is hoped that the project will, in addition, provide the impetus for a pan-European test suite initiative.
TSNLP is funded within the Linguistic Research and Engineering programme of the European Commission (DG XIII) and conducted by a consortium consisting of the University of Essex (project coordinator), the CL department at DFKI Saarbrücken, the Istituto dalle Molle per gli studii semantici e cognitivi (ISSCO), Geneva, and the Common Research Centre of Aerospatiale in Suresnes, France.
Partner
University of Essex (Coordinator), ISSCO, Aerospatiale
