  Quantification of Uncertainty in predictions is a challenging problem. In the
classification settings, although deep learning based models generalize well,
class probabilities often lack reliability. Calibration errors are used to
quantify uncertainty, and several methods exist to minimize calibration error.
We argue that between the choice of having a minimum calibration error on
original distribution which increases across distortions or having a (possibly
slightly higher) calibration error which is constant across distortions, we
prefer the latter
  We hypothesize that the reason for unreliability of deep networks is - The
way neural networks are currently trained, the probabilities do not generalize
across small distortions. We observe that quantile based approaches can
potentially solve this problem. We propose an innovative approach to decouple
the construction of quantile representations from the loss function allowing us
to compute quantile based probabilities without disturbing the original
network. We achieve this by establishing a novel duality property between
quantiles and probabilities, and an ability to obtain quantile probabilities
from any pre-trained classifier.
  While post-hoc calibration techniques successfully minimize calibration
errors, they do not preserve robustness to distortions. We show that, Quantile
probabilities (QuantProb), obtained from Quantile representations, preserve the
calibration errors across distortions, since quantile probabilities generalize
better than the naive Softmax probabilities.
