  Medical data collected for making a diagnostic decision are typically
multi-modal and provide complementary perspectives of a subject. A
computer-aided diagnosis system welcomes multi-modal inputs; however, how to
effectively fuse such multi-modal data is a challenging task and attracts a lot
of attention in the medical research field. In this paper, we propose a
transformer-based framework, called Alifuse, for aligning and fusing
multi-modal medical data. Specifically, we convert images and unstructured and
structured texts into vision and language tokens, and use intramodal and
intermodal attention mechanisms to learn holistic representations of all
imaging and non-imaging data for classification. We apply Alifuse to classify
Alzheimer's disease and obtain state-of-the-art performance on five public
datasets, by outperforming eight baselines. The source code will be available
online later.
