  Electronic health record (EHR) systems contain a wealth of multimodal
clinical data including structured data like clinical codes and unstructured
data such as clinical notes. However, many existing EHR-focused studies has
traditionally either concentrated on an individual modality or merged different
modalities in a rather rudimentary fashion. This approach often results in the
perception of structured and unstructured data as separate entities, neglecting
the inherent synergy between them. Specifically, the two important modalities
contain clinically relevant, inextricably linked and complementary health
information. A more complete picture of a patient's medical history is captured
by the joint analysis of the two modalities of data. Despite the great success
of multimodal contrastive learning on vision-language, its potential remains
under-explored in the realm of multimodal EHR, particularly in terms of its
theoretical understanding. To accommodate the statistical analysis of
multimodal EHR data, in this paper, we propose a novel multimodal feature
embedding generative model and design a multimodal contrastive loss to obtain
the multimodal EHR feature representation. Our theoretical analysis
demonstrates the effectiveness of multimodal learning compared to
single-modality learning and connects the solution of the loss function to the
singular value decomposition of a pointwise mutual information matrix. This
connection paves the way for a privacy-preserving algorithm tailored for
multimodal EHR feature representation learning. Simulation studies show that
the proposed algorithm performs well under a variety of configurations. We
further validate the clinical utility of the proposed algorithm in real-world
EHR data.
