  Successful deployment of Deep Neural Networks (DNNs) requires their
validation with an adequate test set to ensure a sufficient degree of
confidence in test outcomes. Although well-established test adequacy assessment
techniques have been proposed for DNNs, we still need to investigate their
application within a comprehensive methodology for accurately predicting the
fault detection ability of test sets and thus assessing their adequacy. In this
paper, we propose and evaluate TEASMA, a comprehensive and practical
methodology designed to accurately assess the adequacy of test sets for DNNs.
In practice, TEASMA allows engineers to decide whether they can trust
high-accuracy test results and thus validate the DNN before its deployment.
Based on a DNN model's training set, TEASMA provides a procedure to build
accurate DNN-specific prediction models of the Fault Detection Rate (FDR) of a
test set using an existing adequacy metric, thus enabling its assessment. We
evaluated TEASMA with four state-of-the-art test adequacy metrics:
Distance-based Surprise Coverage (DSC), Likelihood-based Surprise Coverage
(LSC), Input Distribution Coverage (IDC), and Mutation Score (MS). Our
extensive empirical evaluation across multiple DNN models and input sets such
as ImageNet, reveals a strong linear correlation between the predicted and
actual FDR values derived from MS, DSC, and IDC, with minimum R^2 values of
0.94 for MS and 0.90 for DSC and IDC. Furthermore, a low average Root Mean
Square Error (RMSE) of 9% between actual and predicted FDR values across all
subjects, when relying on regression analysis and MS, demonstrates the latter's
superior accuracy when compared to DSC and IDC, with RMSE values of 0.17 and
0.18, respectively. Overall, these results suggest that TEASMA provides a
reliable basis for confidently deciding whether to trust test results for DNN
models.
