  The information noise-contrastive estimation (InfoNCE) loss function provides
the basis of many self-supervised deep learning methods due to its strong
empirical results and theoretic motivation. Previous work suggests a supervised
contrastive (SupCon) loss to extend InfoNCE to learn from available class
labels. This SupCon loss has been widely-used due to reports of good empirical
performance. However, in this work we find that the prior SupCon loss
formulation has questionable justification because it can encourage some images
from the same class to repel one another in the learned embedding space. This
problematic intra-class repulsion gets worse as the number of images sharing
one class label increases. We propose the Supervised InfoNCE REvisited
(SINCERE) loss as a theoretically-justified supervised extension of InfoNCE
that eliminates intra-class repulsion. Experiments show that SINCERE leads to
better separation of embeddings from different classes and improves transfer
learning classification accuracy. We additionally utilize probabilistic
modeling to derive an information-theoretic bound that relates SINCERE loss to
the symmeterized KL divergence between data-generating distributions for a
target class and all other classes.
