In many areas of decision making, forecasting is an essential pillar. Consequently, there are many different forecasting methods. According to the “No-Free-Lunch Theorem”, there is no single forecasting method that performs best for all time series. In other words, each method has its advantages and disadvantages depending on the specific use case. Therefore, the choice of the forecasting method remains a mandatory expert task. However, expert knowledge cannot be fully automated. To establish a level playing field for evaluating the performance of time series forecasting methods in a broad setting, we propose Libra, a forecasting benchmark that automatically evaluates and ranks forecasting methods based on their performance in a diverse set of evaluation scenarios. The benchmark comprises four different use cases, each covering 100 heterogeneous time series taken from different domains. The data set was assembled from publicly available time series and was designed to exhibit much higher diversity than existing forecasting competitions. Based on this benchmark, we perform a comprehensive evaluation to compare different existing time series forecasting methods.
Libra is an accepted tool of the Standard Performance Evaluation Corporation (SPEC). The source code can be found on the SPEC tool website or on GitHub.
@inproceedings{bauer2021benchmark,
title = {{Libra: A Benchmark for Time Series Forecasting Methods}},
author = {Bauer, André and Züfle, Marwin and Eismann, Simon and Grohmann, Johannes and Herbst, Nikolas and Kounev, Samuel},
year = {2021},
month = apr,
day = {19--23},
booktitle = {Proceedings of the 12th ACM/SPEC International Conference on Performance Engineering (ICPE)},
publisher = {ACM},
url = {https://dl.acm.org/doi/10.1145/3427921.3450241},
numpages = {12},
selected = {true}
}