  This paper proposes a method for representation learning of multimodal data
using contrastive losses. A traditional approach is to contrast different
modalities to learn the information shared between them. However, that approach
could fail to learn the complementary synergies between modalities that might
be useful for downstream tasks. Another approach is to concatenate all the
modalities into a tuple and then contrast positive and negative tuple
correspondences. However, that approach could consider only the stronger
modalities while ignoring the weaker ones. To address these issues, we propose
a novel contrastive learning objective, TupleInfoNCE. It contrasts tuples based
not only on positive and negative correspondences but also by composing new
negative tuples using modalities describing different scenes. Training with
these additional negatives encourages the learning model to examine the
correspondences among modalities in the same tuple, ensuring that weak
modalities are not ignored. We provide a theoretical justification based on
mutual information for why this approach works, and we propose a sample
optimization algorithm to generate positive and negative samples to maximize
training efficacy. We find that TupleInfoNCE significantly outperforms the
previous state of the arts on three different downstream tasks.
