  Soft targets combined with the cross-entropy loss have shown to improve
generalization performance of deep neural networks on supervised classification
tasks. The standard cross-entropy loss however assumes data to be categorically
distributed, which may often not be the case in practice. In contrast, InfoNCE
does not rely on such an explicit assumption but instead implicitly estimates
the true conditional through negative sampling. Unfortunately, it cannot be
combined with soft targets in its standard formulation, hindering its use in
combination with sophisticated training strategies. In this paper, we address
this limitation by proposing a loss function that is compatible with
probabilistic targets. Our new soft target InfoNCE loss is conceptually simple,
efficient to compute, and can be motivated through the framework of noise
contrastive estimation. Using a toy example, we demonstrate shortcomings of the
categorical distribution assumption of cross-entropy, and discuss implications
of sampling from soft distributions. We observe that soft target InfoNCE
performs on par with strong soft target cross-entropy baselines and outperforms
hard target NLL and InfoNCE losses on popular benchmarks, including ImageNet.
Finally, we provide a simple implementation of our loss, geared towards
supervised classification and fully compatible with deep classification models
trained with cross-entropy.
