  As an effective data augmentation method, Mixup synthesizes an extra amount
of samples through linear interpolations. Despite its theoretical dependency on
data properties, Mixup reportedly performs well as a regularizer and calibrator
contributing reliable robustness and generalization to deep model training. In
this paper, inspired by Universum Learning which uses out-of-class samples to
assist the target tasks, we investigate Mixup from a largely under-explored
perspective - the potential to generate in-domain samples that belong to none
of the target classes, that is, universum. We find that in the framework of
supervised contrastive learning, Mixup-induced universum can serve as
surprisingly high-quality hard negatives, greatly relieving the need for large
batch sizes in contrastive learning. With these findings, we propose
Universum-inspired supervised Contrastive learning (UniCon), which incorporates
Mixup strategy to generate Mixup-induced universum as universum negatives and
pushes them apart from anchor samples of the target classes. We extend our
method to the unsupervised setting, proposing Unsupervised Universum-inspired
contrastive model (Un-Uni). Our approach not only improves Mixup with hard
labels, but also innovates a novel measure to generate universum data. With a
linear classifier on the learned representations, UniCon shows state-of-the-art
performance on various datasets. Specially, UniCon achieves 81.7% top-1
accuracy on CIFAR-100, surpassing the state of art by a significant margin of
5.2% with a much smaller batch size, typically, 256 in UniCon vs. 1024 in
SupCon using ResNet-50. Un-Uni also outperforms SOTA methods on CIFAR-100. The
code of this paper is released on https://github.com/hannaiiyanggit/UniCon.
