  Building scalable models to learn from diverse, multimodal data remains an
open challenge. For vision-language data, the dominant approaches are based on
contrastive learning objectives that train a separate encoder for each
modality. While effective, contrastive learning approaches introduce sampling
bias depending on the data augmentations used, which can degrade performance on
downstream tasks. Moreover, these methods are limited to paired image-text
data, and cannot leverage widely-available unpaired data. In this paper, we
investigate whether a large multimodal model trained purely via masked token
prediction, without using modality-specific encoders or contrastive learning,
can learn transferable representations for downstream tasks. We propose a
simple and scalable network architecture, the Multimodal Masked Autoencoder
(M3AE), which learns a unified encoder for both vision and language data via
masked token prediction. We provide an empirical study of M3AE trained on a
large-scale image-text dataset, and find that M3AE is able to learn
generalizable representations that transfer well to downstream tasks.
Surprisingly, we find that M3AE benefits from a higher text mask ratio
(50-90%), in contrast to BERT whose standard masking ratio is 15%, due to the
joint training of two data modalities. We also provide qualitative analysis
showing that the learned representation incorporates meaningful information
from both image and language. Lastly, we demonstrate the scalability of M3AE
with larger model size and training time, and its flexibility to train on both
paired image-text data as well as unpaired data.
