  Due to the black box nature of deep neural networks (DNN), the continuous
validation of DNN during operation is challenging with the absence of a human
monitor. As a result this makes it difficult for developers and regulators to
gain confidence in the deployment of autonomous systems employing DNN. It is
critical for safety during operation to know when DNN's predictions are
trustworthy or suspicious. With the absence of a human monitor, the basic
approach is to use the model's output confidence score to assess if predictions
are trustworthy or suspicious. However, the model's confidence score is a
result of computations coming from a black box, therefore lacks transparency
and makes it challenging to automatedly credit trustworthiness to predictions.
We introduce the trustworthiness score (TS), a simple metric that provides a
more transparent and effective way of providing confidence in DNN predictions
compared to model's confidence score. The metric quantifies the trustworthiness
in a prediction by checking for the existence of certain features in the
predictions made by the DNN. We also use the underlying idea of the TS metric,
to provide a suspiciousness score (SS) in the overall input frame to help in
the detection of suspicious frames where false negatives exist. We conduct a
case study using YOLOv5 on persons detection to demonstrate our method and
usage of TS and SS. The case study shows that using our method consistently
improves the precision of predictions compared to relying on model confidence
score alone, for both 1) approving of trustworthy predictions (~20%
improvement) and 2) detecting suspicious frames (~5% improvement).
