  Machine learning plays an increasingly significant role in many aspects of
our lives (including medicine, transportation, security, justice and other
domains), making the potential consequences of false predictions increasingly
devastating. These consequences may be mitigated if we can automatically flag
such false predictions and potentially assign them to alternative, more
reliable mechanisms, that are possibly more costly and involve human attention.
This suggests the task of detecting errors, which we tackle in this paper for
the case of visual classification. To this end, we propose a novel approach for
classification confidence estimation. We apply a set of semantics-preserving
image transformations to the input image, and show how the resulting image sets
can be used to estimate confidence in the classifier's prediction. We
demonstrate the potential of our approach by extensively evaluating it on a
wide variety of classifier architectures and datasets, including
ResNext/ImageNet, achieving state of the art performance. This paper
constitutes a significant revision of our earlier work in this direction (Bahat
& Shakhnarovich, 2018).
