  A wide breadth of research has devised data augmentation approaches that can
improve both accuracy and generalization performance for neural networks.
However, augmented data can end up being far from the clean training data and
what is the appropriate label is less clear. Despite this, most existing work
simply uses one-hot labels for augmented data. In this paper, we show re-using
one-hot labels for highly distorted data might run the risk of adding noise and
degrading accuracy and calibration. To mitigate this, we propose a generic
method AutoLabel to automatically learn the confidence in the labels for
augmented data, based on the transformation distance between the clean
distribution and augmented distribution. AutoLabel is built on label smoothing
and is guided by the calibration-performance over a hold-out validation set. We
successfully apply AutoLabel to three different data augmentation techniques:
the state-of-the-art RandAug, AugMix, and adversarial training. Experiments on
CIFAR-10, CIFAR-100 and ImageNet show that AutoLabel significantly improves
existing data augmentation techniques over models' calibration and accuracy,
especially under distributional shift.
