  Electronic Health Record (EHR) datasets from Intensive Care Units (ICU)
contain a diverse set of data modalities. While prior works have successfully
leveraged multiple modalities in supervised settings, we apply advanced
self-supervised multi-modal contrastive learning techniques to ICU data,
specifically focusing on clinical notes and time-series for clinically relevant
online prediction tasks. We introduce a loss function Multi-Modal Neighborhood
Contrastive Loss (MM-NCL), a soft neighborhood function, and showcase the
excellent linear probe and zero-shot performance of our approach.
