  Promptly discovering unknown network attacks is critical for reducing the
risk of major loss imposed on system or equipment. This paper aims to develop
an open-set intrusion detection model to classify known attacks as well as
inferring unknown ones. To achieve this, we employ OpenMax and variational
autoencoder to propose a dual detection model, VAEMax. First, we extract flow
payload feature based on one-dimensional convolutional neural network. Then,
the OpenMax is used to classify flows, during which some unknown attacks can be
detected, while the rest are misclassified into a certain class of known flows.
Finally, use VAE to perform secondary detection on each class of flows, and
determine whether the flow is an unknown attack based on the reconstruction
loss. Experiments performed on dataset CIC-IDS2017 and CSE-CIC-IDS2018 show our
approach is better than baseline models and can be effectively applied to
realistic network environments.
