  Uncertainty estimates help to identify ambiguous, novel, or anomalous inputs,
but the reliable quantification of uncertainty has proven to be challenging for
modern deep networks. In order to improve uncertainty estimation, we propose
On-Manifold Adversarial Data Augmentation or OMADA, which specifically attempts
to generate the most challenging examples by following an on-manifold
adversarial attack path in the latent space of an autoencoder-based generative
model that closely approximates decision boundaries between two or more
classes. On a variety of datasets as well as on multiple diverse network
architectures, OMADA consistently yields more accurate and better calibrated
classifiers than baseline models, and outperforms competing approaches such as
Mixup, as well as achieving similar performance to (at times better than)
post-processing calibration methods such as temperature scaling. Variants of
OMADA can employ different sampling schemes for ambiguous on-manifold examples
based on the entropy of their estimated soft labels, which exhibit specific
strengths for generalization, calibration of predicted uncertainty, or
detection of out-of-distribution inputs.
