  Deep Neural Networks (DNN) are increasingly used as components of larger
software systems that need to process complex data, such as images, written
texts, audio/video signals. DNN predictions cannot be assumed to be always
correct for several reasons, among which the huge input space that is dealt
with, the ambiguity of some inputs data, as well as the intrinsic properties of
learning algorithms, which can provide only statistical warranties. Hence,
developers have to cope with some residual error probability. An architectural
pattern commonly adopted to manage failure-prone components is the supervisor,
an additional component that can estimate the reliability of the predictions
made by untrusted (e.g., DNN) components and can activate an automated healing
procedure when these are likely to fail, ensuring that the Deep Learning based
System (DLS) does not cause damages, despite its main functionality being
suspended.
  In this paper, we consider DLS that implement a supervisor by means of
uncertainty estimation. After overviewing the main approaches to uncertainty
estimation and discussing their pros and cons, we motivate the need for a
specific empirical assessment method that can deal with the experimental
setting in which supervisors are used, where the accuracy of the DNN matters
only as long as the supervisor lets the DLS continue to operate. Then we
present a large empirical study conducted to compare the alternative approaches
to uncertainty estimation. We distilled a set of guidelines for developers that
are useful to incorporate a supervisor based on uncertainty monitoring into a
DLS.
