  In this work, we propose data augmentation methods for embeddings from
pre-trained deep learning models that take a weighted combination of a pair of
input embeddings, as inspired by Mixup, and combine such augmentation with
extra label softening. These methods are shown to significantly increase
classification accuracy, reduce training time, and improve confidence
calibration of a downstream model that is trained with them. As a result of
such improved confidence calibration, the model output can be more intuitively
interpreted and used to accurately identify out-of-distribution data by
applying an appropriate confidence threshold to model predictions. The
identified out-of-distribution data can then be prioritized for labeling, thus
focusing labeling effort on data that is more likely to boost model
performance. These findings, we believe, lay a solid foundation for improving
the classification performance and calibration of models that use pre-trained
embeddings as input and provide several benefits that prove extremely useful in
a production-level deep learning system.
