A Re-evaluation of Intrusion Detection Accuracy: Alternative Evaluation Strategy

This work tries to evaluate the existing approaches used to benchmark the performance of machine learning models applied to network-based intrusion detection systems (NIDS). First, we demonstrate that we can reach a very high accuracy with most of the traditional machine learning and deep learning models by using the existing performance evaluation strategy. It just requires the right hyperparameter tuning to outperform the existing reported accuracy results in deep learning models. We further question the value of the existing evaluation methods in which the same datasets are used for training and testing the models. We are proposing the use of an alternative strategy that aims to evaluate the practicality and the performance of the models and datasets as well. In this approach, different datasets with compatible sets of features are used for training and testing. When we evaluate the models that we created with the proposed strategy, we demonstrate that the performance is very bad. Thus, models have no practical usage, and it performs based on a pure randomness. This research is important for security-based machine learning applications to re-think about the datasets and the model's quality.