  Multifold observations are common for different data modalities, e.g., a 3D
shape can be represented by multi-view images and an image can be described
with different captions. Existing cross-modal contrastive representation
learning (XM-CLR) methods such as CLIP are not fully suitable for multifold
data as they only consider one positive pair and treat other pairs as negative
when computing the contrastive loss. In this paper, we propose MXM-CLR, a
unified framework for contrastive learning of multifold cross-modal
representations. MXM-CLR explicitly models and learns the relationships between
multifold observations of instances from different modalities for more
comprehensive representation learning. The key of MXM-CLR is a novel
multifold-aware hybrid loss which considers multiple positive observations when
computing the hard and soft relationships for the cross-modal data pairs. We
conduct quantitative and qualitative comparisons with SOTA baselines for
cross-modal retrieval tasks on the Text2Shape and Flickr30K datasets. We also
perform extensive evaluations on the adaptability and generalizability of
MXM-CLR, as well as ablation studies on the loss design and effects of batch
sizes. The results show the superiority of MXM-CLR in learning better
representations for the multifold data. The code is available at
https://github.com/JLU-ICL/MXM-CLR.
