  Modern Web systems such as social media and e-commerce contain rich contents
expressed in images and text. Leveraging information from multi-modalities can
improve the performance of machine learning tasks such as classification and
recommendation. In this paper, we propose the Cross-Modality Attention
Contrastive Language-Image Pre-training (CMA-CLIP), a new framework which
unifies two types of cross-modality attentions, sequence-wise attention and
modality-wise attention, to effectively fuse information from image and text
pairs. The sequence-wise attention enables the framework to capture the
fine-grained relationship between image patches and text tokens, while the
modality-wise attention weighs each modality by its relevance to the downstream
tasks. In addition, by adding task specific modality-wise attentions and
multilayer perceptrons, our proposed framework is capable of performing
multi-task classification with multi-modalities.
  We conduct experiments on a Major Retail Website Product Attribute (MRWPA)
dataset and two public datasets, Food101 and Fashion-Gen. The results show that
CMA-CLIP outperforms the pre-trained and fine-tuned CLIP by an average of 11.9%
in recall at the same level of precision on the MRWPA dataset for multi-task
classification. It also surpasses the state-of-the-art method on Fashion-Gen
Dataset by 5.5% in accuracy and achieves competitive performance on Food101
Dataset. Through detailed ablation studies, we further demonstrate the
effectiveness of both cross-modality attention modules and our method's
robustness against noise in image and text inputs, which is a common challenge
in practice.
