  Training multimodal foundation models is challenging due to the limited
availability of multimodal datasets. While many public datasets pair images
with text, few combine images with audio or text with audio. Even rarer are
datasets that align all three modalities at once. Critical domains such as
healthcare, infrastructure, or transportation are particularly affected by
missing modalities. This makes it difficult to integrate all modalities into a
large pre-trained neural network that can be used out-of-the-box or fine-tuned
for different downstream tasks. We introduce LoReTTa (Linking mOdalities with a
tRansitive and commutativE pre-Training sTrAtegy) to address this understudied
problem. Our self-supervised framework unifies causal modeling and masked
modeling with the rules of commutativity and transitivity. This allows us to
transition within and between modalities. As a result, our pre-trained models
are better at exploring the true underlying joint probability distribution.
Given a dataset containing only the disjoint combinations (A, B) and (B, C),
LoReTTa can model the relation A <-> C with A <-> B <-> C. In particular, we
show that a transformer pre-trained with LoReTTa can handle any mixture of
modalities at inference time, including the never-seen pair (A, C) and the
triplet (A, B, C). We extensively evaluate our approach on a synthetic,
medical, and reinforcement learning dataset. Across different domains, our
universal multimodal transformer consistently outperforms strong baselines such
as GPT, BERT, and CLIP on tasks involving the missing modality tuple.
