  There has been a large number of studies in interpretable and explainable ML
for cybersecurity, in particular, for intrusion detection. Many of these
studies have significant amount of overlapping and repeated evaluations and
analysis. At the same time, these studies overlook crucial model, data,
learning process, and utility related issues and many times completely
disregard them. These issues include the use of overly complex and opaque ML
models, unaccounted data imbalances and correlated features, inconsistent
influential features across different explanation methods, the inconsistencies
stemming from the constituents of a learning process, and the implausible
utility of explanations. In this work, we empirically demonstrate these issues,
analyze them and propose practical solutions in the context of feature-based
model explanations. Specifically, we advise avoiding complex opaque models such
as Deep Neural Networks and instead using interpretable ML models such as
Decision Trees as the available intrusion datasets are not difficult for such
interpretable models to classify successfully. Then, we bring attention to the
binary classification metrics such as Matthews Correlation Coefficient (which
are well-suited for imbalanced datasets. Moreover, we find that feature-based
model explanations are most often inconsistent across different settings. In
this respect, to further gauge the extent of inconsistencies, we introduce the
notion of cross explanations which corroborates that the features that are
determined to be impactful by one explanation method most often differ from
those by another method. Furthermore, we show that strongly correlated data
features and the constituents of a learning process, such as hyper-parameters
and the optimization routine, become yet another source of inconsistent
explanations. Finally, we discuss the utility of feature-based explanations.
