  Long-tailed distributions frequently emerge in real-world data, where a large
number of minority categories contain a limited number of samples. Such
imbalance issue considerably impairs the performance of standard supervised
learning algorithms, which are mainly designed for balanced training sets.
Recent investigations have revealed that supervised contrastive learning
exhibits promising potential in alleviating the data imbalance. However, the
performance of supervised contrastive learning is plagued by an inherent
challenge: it necessitates sufficiently large batches of training data to
construct contrastive pairs that cover all categories, yet this requirement is
difficult to meet in the context of class-imbalanced data. To overcome this
obstacle, we propose a novel probabilistic contrastive (ProCo) learning
algorithm that estimates the data distribution of the samples from each class
in the feature space, and samples contrastive pairs accordingly. In fact,
estimating the distributions of all classes using features in a small batch,
particularly for imbalanced data, is not feasible. Our key idea is to introduce
a reasonable and simple assumption that the normalized features in contrastive
learning follow a mixture of von Mises-Fisher (vMF) distributions on unit
space, which brings two-fold benefits. First, the distribution parameters can
be estimated using only the first sample moment, which can be efficiently
computed in an online manner across different batches. Second, based on the
estimated distribution, the vMF distribution allows us to sample an infinite
number of contrastive pairs and derive a closed form of the expected
contrastive loss for efficient optimization. Our code is available at
https://github.com/LeapLabTHU/ProCo.
