  The integration of multimodal Electronic Health Records (EHR) data has
significantly improved clinical predictive capabilities. Leveraging clinical
notes and multivariate time-series EHR, existing models often lack the medical
context relevent to clinical tasks, prompting the incorporation of external
knowledge, particularly from the knowledge graph (KG). Previous approaches with
KG knowledge have primarily focused on structured knowledge extraction,
neglecting unstructured data modalities and semantic high dimensional medical
knowledge. In response, we propose REALM, a Retrieval-Augmented Generation
(RAG) driven framework to enhance multimodal EHR representations that address
these limitations. Firstly, we apply Large Language Model (LLM) to encode long
context clinical notes and GRU model to encode time-series EHR data. Secondly,
we prompt LLM to extract task-relevant medical entities and match entities in
professionally labeled external knowledge graph (PrimeKG) with corresponding
medical knowledge. By matching and aligning with clinical standards, our
framework eliminates hallucinations and ensures consistency. Lastly, we propose
an adaptive multimodal fusion network to integrate extracted knowledge with
multimodal EHR data. Our extensive experiments on MIMIC-III mortality and
readmission tasks showcase the superior performance of our REALM framework over
baselines, emphasizing the effectiveness of each module. REALM framework
contributes to refining the use of multimodal EHR data in healthcare and
bridging the gap with nuanced medical context essential for informed clinical
predictions.
