  Contrastive Language Image Pre-training (CLIP) has recently demonstrated
success across various tasks due to superior feature representation empowered
by image-text contrastive learning. However, the instance discrimination method
used by CLIP can hardly encode the semantic structure of training data. To
handle this limitation, cluster discrimination has been proposed through
iterative cluster assignment and classification. Nevertheless, most cluster
discrimination approaches only define a single pseudo-label for each image,
neglecting multi-label signals in the image. In this paper, we propose a novel
Multi-Label Cluster Discrimination method named MLCD to enhance representation
learning. In the clustering step, we first cluster the large-scale LAION-400M
dataset into one million centers based on off-the-shelf embedding features.
Considering that natural images frequently contain multiple visual objects or
attributes, we select the multiple closest centers as auxiliary class labels.
In the discrimination step, we design a novel multi-label classification loss,
which elegantly separates losses from positive classes and negative classes,
and alleviates ambiguity on decision boundary. We validate the proposed
multi-label cluster discrimination method with experiments on different scales
of models and pre-training datasets. Experimental results show that our method
achieves state-of-the-art performance on multiple downstream tasks including
linear probe, zero-shot classification, and image-text retrieval.
