  Medical datasets and especially biobanks, often contain extensive tabular
data with rich clinical information in addition to images. In practice,
clinicians typically have less data, both in terms of diversity and scale, but
still wish to deploy deep learning solutions. Combined with increasing medical
dataset sizes and expensive annotation costs, the necessity for unsupervised
methods that can pretrain multimodally and predict unimodally has risen.
  To address these needs, we propose the first self-supervised contrastive
learning framework that takes advantage of images and tabular data to train
unimodal encoders. Our solution combines SimCLR and SCARF, two leading
contrastive learning strategies, and is simple and effective. In our
experiments, we demonstrate the strength of our framework by predicting risks
of myocardial infarction and coronary artery disease (CAD) using cardiac MR
images and 120 clinical features from 40,000 UK Biobank subjects. Furthermore,
we show the generalizability of our approach to natural images using the DVM
car advertisement dataset.
  We take advantage of the high interpretability of tabular data and through
attribution and ablation experiments find that morphometric tabular features,
describing size and shape, have outsized importance during the contrastive
learning process and improve the quality of the learned embeddings. Finally, we
introduce a novel form of supervised contrastive learning, label as a feature
(LaaF), by appending the ground truth label as a tabular feature during
multimodal pretraining, outperforming all supervised contrastive baselines.
