  Only increasing accuracy without considering uncertainty may negatively
impact Deep Neural Network (DNN) decision-making and decrease its reliability.
This paper proposes five combined preprocessing and post-processing methods for
time-series binary classification problems that simultaneously increase the
accuracy and reliability of DNN outputs applied in a 5G UAV security dataset.
These techniques use DNN outputs as input parameters and process them in
different ways. Two methods use a well-known Machine Learning (ML) algorithm as
a complement, and the other three use only confidence values that the DNN
estimates. We compare seven different metrics, such as the Expected Calibration
Error (ECE), Maximum Calibration Error (MCE), Mean Confidence (MC), Mean
Accuracy (MA), Normalized Negative Log Likelihood (NLL), Brier Score Loss
(BSL), and Reliability Score (RS) and the tradeoffs between them to evaluate
the proposed hybrid algorithms. First, we show that the eXtreme Gradient
Boosting (XGB) classifier might not be reliable for binary classification under
the conditions this work presents. Second, we demonstrate that at least one of
the potential methods can achieve better results than the classification in the
DNN softmax layer. Finally, we show that the prospective methods may improve
accuracy and reliability with better uncertainty calibration based on the
assumption that the RS determines the difference between MC and MA metrics, and
this difference should be zero to increase reliability. For example, Method 3
presents the best RS of 0.65 even when compared to the XGB classifier, which
achieves RS of 7.22.
