  Self-supervised learning is crucial for clinical imaging applications, given
the lack of explicit labels in healthcare. However, conventional approaches
that rely on precise vision-language alignment are not always feasible in
complex clinical imaging modalities, such as cardiac magnetic resonance (CMR).
CMR provides a comprehensive visualization of cardiac anatomy, physiology, and
microstructure, making it challenging to interpret. Additionally, CMR reports
require synthesizing information from sequences of images and different views,
resulting in potentially weak alignment between the study and diagnosis report
pair. To overcome these challenges, we propose \textbf{CMRformer}, a multimodal
learning framework to jointly learn sequences of CMR images and associated
cardiologist's reports. Moreover, one of the major obstacles to improving CMR
study is the lack of large, publicly available datasets. To bridge this gap, we
collected a large \textbf{CMR dataset}, which consists of 13,787 studies from
clinical cases. By utilizing our proposed CMRformer and our collected dataset,
we achieved remarkable performance in real-world clinical tasks, such as CMR
image retrieval and diagnosis report retrieval. Furthermore, the learned
representations are evaluated to be practically helpful for downstream
applications, such as disease classification. Our work could potentially
expedite progress in the CMR study and lead to more accurate and effective
diagnosis and treatment.
