Forecasting IT Security Vulnerabilities - An Empirical Analysis

Today, organizations must deal with a plethora of IT security threats and to ensure smooth and uninterrupted business operations, firms are challenged to predict the volume of IT security vulnerabilities and allocate resources for fixing them. This challenge requires decision makers to assess which system or software packages are prone to vulnerabilities, how many post-release vulnerabilities can be expected to occur during a certain period of time, and what impact exploits might have. Substantial research has been dedicated to techniques that analyze source code and detect security vulnerabilities. However, only limited research has focused on forecasting security vulnerabilities that are detected and reported after the release of software. To address this shortcoming, we apply established methodologies which are capable of forecasting events exhibiting specific time series characteristics of security vulnerabilities, i.e., rareness of occurrence, volatility, non-stationarity, and seasonality. Based on a dataset taken from the National Vulnerability Database (NVD), we use the Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) to measure the forecasting accuracy of single, double, and triple exponential smoothing methodologies, Croston's methodology, ARIMA, and a neural network-based approach. We analyze the impact of the applied forecasting methodology on the prediction accuracy with regard to its robustness along the dimensions of the examined system and software package "operating systems", "browsers" and "office solutions" and the applied metrics. To the best of our knowledge, this study is the first to analyze the effect of forecasting methodologies and to apply metrics that are suitable in this context. Our results show that the optimal forecasting methodology depends on the software or system package, as some methodologies perform poorly in the context of IT security vulnerabilities, that absolute metrics can cover the actual prediction error precisely, and that the prediction accuracy is robust within the two applied forecasting-error metrics.