  Sensory input from multiple sources is crucial for robust and coherent human
perception. Different sources contribute complementary explanatory factors.
Similarly, research studies often collect multimodal imaging data, each of
which can provide shared and unique information. This observation motivated the
design of powerful multimodal self-supervised representation-learning
algorithms. In this paper, we unify recent work on multimodal self-supervised
learning under a single framework. Observing that most self-supervised methods
optimize similarity metrics between a set of model components, we propose a
taxonomy of all reasonable ways to organize this process. We first evaluate
models on toy multimodal MNIST datasets and then apply them to a multimodal
neuroimaging dataset with Alzheimer's disease patients. We find that (1)
multimodal contrastive learning has significant benefits over its unimodal
counterpart, (2) the specific composition of multiple contrastive objectives is
critical to performance on a downstream task, (3) maximization of the
similarity between representations has a regularizing effect on a neural
network, which can sometimes lead to reduced downstream performance but still
reveal multimodal relations. Results show that the proposed approach
outperforms previous self-supervised encoder-decoder methods based on canonical
correlation analysis (CCA) or the mixture-of-experts multimodal variational
autoEncoder (MMVAE) on various datasets with a linear evaluation protocol.
Importantly, we find a promising solution to uncover connections between
modalities through a jointly shared subspace that can help advance work in our
search for neuroimaging biomarkers.
