  Computer-assisted diagnostic and prognostic systems of the future should be
capable of simultaneously processing multimodal data. Multimodal deep learning
(MDL), which involves the integration of multiple sources of data, such as
images and text, has the potential to revolutionize the analysis and
interpretation of biomedical data. However, it only caught researchers'
attention recently. To this end, there is a critical need to conduct a
systematic review on this topic, identify the limitations of current work, and
explore future directions. In this scoping review, we aim to provide a
comprehensive overview of the current state of the field and identify key
concepts, types of studies, and research gaps with a focus on biomedical images
and texts joint learning, mainly because these two were the most commonly
available data types in MDL research. This study reviewed the current uses of
multimodal deep learning on five tasks: (1) Report generation, (2) Visual
question answering, (3) Cross-modal retrieval, (4) Computer-aided diagnosis,
and (5) Semantic segmentation. Our results highlight the diverse applications
and potential of MDL and suggest directions for future research in the field.
We hope our review will facilitate the collaboration of natural language
processing (NLP) and medical imaging communities and support the next
generation of decision-making and computer-assisted diagnostic system
development.
