  While deep neural network (DNN)-based perception models are useful for many
applications, these models are black boxes and their outputs are not yet well
understood. To confidently enable a real-world, decision-making system to
utilize such a perception model without human intervention, we must enable the
system to reason about the perception model's level of competency and respond
appropriately when the model is incompetent. In order for the system to make an
intelligent decision about the appropriate action when the model is
incompetent, it would be useful for the system to understand why the model is
incompetent. We explore five novel methods for identifying regions in the input
image contributing to low model competency, which we refer to as image
cropping, segment masking, pixel perturbation, competency gradients, and
reconstruction loss. We assess the ability of these five methods to identify
unfamiliar objects, recognize regions associated with unseen classes, and
identify unexplored areas in an environment. We find that the competency
gradients and reconstruction loss methods show great promise in identifying
regions associated with low model competency, particularly when aspects of the
image that are unfamiliar to the perception model are causing this reduction in
competency. Both of these methods boast low computation times and high levels
of accuracy in detecting image regions that are unfamiliar to the model,
allowing them to provide potential utility in decision-making pipelines. The
code for reproducing our methods and results is available on GitHub:
https://github.com/sarapohland/explainable-competency.
