  Machine Learning (ML)-based network intrusion detection systems bring many
benefits for enhancing the cybersecurity posture of an organisation. Many
systems have been designed and developed in the research community, often
achieving a close to perfect detection rate when evaluated using synthetic
datasets. However, the high number of academic research has not often
translated into practical deployments. There are several causes contributing
towards the wide gap between research and production, such as the limited
ability of comprehensive evaluation of ML models and lack of understanding of
internal ML operations. This paper tightens the gap by evaluating the
generalisability of a common feature set to different network environments and
attack scenarios. Therefore, two feature sets (NetFlow and CICFlowMeter) have
been evaluated in terms of detection accuracy across three key datasets, i.e.,
CSE-CIC-IDS2018, BoT-IoT, and ToN-IoT. The results show the superiority of the
NetFlow feature set in enhancing the ML models detection accuracy of various
network attacks. In addition, due to the complexity of the learning models,
SHapley Additive exPlanations (SHAP), an explainable AI methodology, has been
adopted to explain and interpret the classification decisions of ML models. The
Shapley values of two common feature sets have been analysed across multiple
datasets to determine the influence contributed by each feature towards the
final ML prediction.
