  As the volume of Electronic Health Records (EHR) sharply grows, there has
been emerging interest in learning the representation of EHR for healthcare
applications. Representation learning of EHR requires appropriate modeling of
the two dominant modalities in EHR: structured data and unstructured text. In
this paper, we present MedGTX, a pre-trained model for multi-modal
representation learning of the structured and textual EHR data. MedGTX uses a
novel graph encoder to exploit the graphical nature of structured EHR data, and
a text encoder to handle unstructured text, and a cross-modal encoder to learn
a joint representation space. We pre-train our model through four proxy tasks
on MIMIC-III, an open-source EHR data, and evaluate our model on two clinical
benchmarks and three novel downstream tasks which tackle real-world problems in
EHR data. The results consistently show the effectiveness of pre-training the
model for joint representation of both structured and unstructured information
from EHR. Given the promising performance of MedGTX, we believe this work opens
a new door to jointly understanding the two fundamental modalities of EHR data.
