  The reliable measurement of confidence in classifiers' predictions is very
important for many applications and is, therefore, an important part of
classifier design. Yet, although deep learning has received tremendous
attention in recent years, not much progress has been made in quantifying the
prediction confidence of neural network classifiers. Bayesian models offer a
mathematically grounded framework to reason about model uncertainty, but
usually come with prohibitive computational costs. In this paper we propose a
simple, scalable method to achieve a reliable confidence score, based on the
data embedding derived from the penultimate layer of the network. We
investigate two ways to achieve desirable embeddings, by using either a
distance-based loss or Adversarial Training. We then test the benefits of our
method when used for classification error prediction, weighting an ensemble of
classifiers, and novelty detection. In all tasks we show significant
improvement over traditional, commonly used confidence scores.
