Publication
Evaluating Forecasting Methods by Considering Different Accuracy Measures
Nijat Mehdiyev; David Enke; Peter Fettke; Peter Loos
In: Procedia Computer Science. Complex Adaptive Systems (CAS-2016), Elsevier, 2016.
Abstract
Choosing the appropriate forecasting technique to employ is a challenging issue and requires a comprehensive analysis of empirical results. Recent research findings reveal that the performance evaluation of forecasting models depends on the accuracy measures adopted. Some methods indicate superior performance when error-based metrics are used, while others perform better when precision values are adopted as accuracy measures. As scholars tend to use a smaller subset of accuracy metrics to assess the performance of forecasting models, there is a need for a concept of multiple accuracy dimensions to assure the robustness of evaluation. Therefore, the main purpose of this paper is to propose a decision-making model that allows researchers to identify the superiority of a forecasting technique over another by considering several accuracy metrics concurrently. A multi-criteria decision analysis approach, namely the preference ranking organization method for enrichment evaluation (PROMETHEE), was adopted to solve this problem. Bayesian Networks, Artificial Neural Networks, SVMs, Logistic Regression, and several Rule and Tree-based forecasting approaches were included in the analysis. After introducing a detailed description of accuracy measures, the performance of the prediction models is evaluated using a chosen dataset from the UCI Machine Learning Repository.