  Language modeling have shown impressive progress in generating compelling
text with good accuracy and high semantic coherence. An interesting research
direction is to augment these powerful models for specific applications using
contextual information. In this work, we explore multi-modal language modeling
for healthcare applications. We are interested in outcome prediction and
patient triage in hospital emergency department based on text information in
chief complaints and vital signs recorded at triage. We adapt Perceiver - a
modality-agnostic transformer-based model that has shown promising results in
several applications. Since vital-sign modality is represented in tabular
format, we modified Perceiver position encoding to ensure permutation
invariance. We evaluated the multi-modal language model for the task of
diagnosis code prediction using MIMIC-IV ED dataset on 120K visits. In the
experimental analysis, we show that mutli-modality improves the prediction
performance compared with models trained solely on text or vital signs. We
identified disease categories for which multi-modality leads to performance
improvement and show that for these categories, vital signs have added
predictive power. By analyzing the cross-attention layer, we show how
multi-modality contributes to model predictions. This work gives interesting
insights on the development of multi-modal language models for healthcare
applications.
