  Multi-label image classification is a foundational topic in various domains.
Multimodal learning approaches have recently achieved outstanding results in
image representation and single-label image classification. For instance,
Contrastive Language-Image Pretraining (CLIP) demonstrates impressive
image-text representation learning abilities and is robust to natural
distribution shifts. This success inspires us to leverage multimodal learning
for multi-label classification tasks, and benefit from contrastively learnt
pretrained models. We propose the Multimodal Multi-label Image Classification
(MuMIC) framework, which utilizes a hardness-aware tempered sigmoid based
Binary Cross Entropy loss function, thus enables the optimization on
multi-label objectives and transfer learning on CLIP. MuMIC is capable of
providing high classification performance, handling real-world noisy data,
supporting zero-shot predictions, and producing domain-specific image
embeddings. In this study, a total of 120 image classes are defined, and more
than 140K positive annotations are collected on approximately 60K Booking.com
images. The final MuMIC model is deployed on Booking.com Content Intelligence
Platform, and it outperforms other state-of-the-art models with 85.6% GAP@10
and 83.8% GAP on all 120 classes, as well as a 90.1% macro mAP score across 32
majority classes. We summarize the modeling choices which are extensively
tested through ablation studies. To the best of our knowledge, we are the first
to adapt contrastively learnt multimodal pretraining for real-world multi-label
image classification problems, and the innovation can be transferred to other
domains.
