An intrusion detection system built on deep learning.

As it stands, this project is intended to be a proof-of-concept for an intrusion detection system, and not an intrusion prevention system. Further work could extend this project to an IPS or a near-realtime IDS using a high-performance, distributed computing environment.

We've used the KDDCup99 dataset from OpenML.org. There are several known issues surrounding this dataset; please see the data/README.md file for more information.

We trained three neural networks, as well as a logistic regression classifier for stacking. All models were built with Keras on a Tensorflow backend. The architectures can be found in each network's respective development notebook in develop/. We trained each model using the Adam optimizer with a batch size of 32 and Keras defaults for beta hyperparameters. We also used early stopping monitoring validation accuracy with a patience of three epochs. Other hyperparameters differed for each model, and are shown below:

We monitored validation accuracy to checkpoint the best model from each training run, then chose the best model with respect to validation accuracy for each architecture from all of our training runs. Note that in general, it is not advisable to use accuracy when working with a highly imbalanced dataset like KDDCup99. However, since we are using an ensemble, we want to encourage the base models to develop their own strengths and weaknesses. Using accuracy as a model selection criterion allows for this by ignoring the class imbalance in its selection process.

Below are our results on the test set. Binary results are determined by converting predictions and labels to a binary classification of normal vs. malicious connection attempts. Note that the result for binary recall is not a realistic estimate of real world performance, while precision estimates are more realistic. This is due to deficiencies of the KDDCup99 dataset (as noted here). We've included a standard softmax classifier as a baseline.

Here are the binary results by class. See our blog post for an interactive version.

We were unaware of the existence of an updated intrusion detection benchmarking dataset until the end of this project. Now that we know that NSL-KDD exists, we will be reiterating our method on it. We will then perform a more complete literature search and compare our method with state of the art results. Stay tuned!