  Medical vision-and-language pre-training provides a feasible solution to
extract effective vision-and-language representations from medical images and
texts. However, few studies have been dedicated to this field to facilitate
medical vision-and-language understanding. In this paper, we propose a
self-supervised learning paradigm with multi-modal masked autoencoders
(M$^3$AE), which learn cross-modal domain knowledge by reconstructing missing
pixels and tokens from randomly masked images and texts. There are three key
designs to make this simple approach work. First, considering the different
information densities of vision and language, we adopt different masking ratios
for the input image and text, where a considerably larger masking ratio is used
for images. Second, we use visual and textual features from different layers to
perform the reconstruction to deal with different levels of abstraction in
visual and language. Third, we develop different designs for vision and
language decoders (i.e., a Transformer for vision and a multi-layer perceptron
for language). To perform a comprehensive evaluation and facilitate further
research, we construct a medical vision-and-language benchmark including three
tasks. Experimental results demonstrate the effectiveness of our approach,
where state-of-the-art results are achieved on all downstream tasks. Besides,
we conduct further analysis to better verify the effectiveness of different
components of our approach and various settings of pre-training. The source
code is available at~\url{https://github.com/zhjohnchan/M3AE}.
