  Humans make accurate decisions by interpreting complex data from multiple
sources. Medical diagnostics, in particular, often hinge on human
interpretation of multi-modal information. In order for artificial intelligence
to make progress in automated, objective, and accurate diagnosis and prognosis,
methods to fuse information from multiple medical imaging modalities are
required. However, combining information from multiple data sources has several
challenges, as current deep learning architectures lack the ability to extract
useful representations from multimodal information, and often simple
concatenation is used to fuse such information. In this work, we propose
Multimodal DenseNet, a novel architecture for fusing multimodal data. Instead
of focusing on concatenation or early and late fusion, our proposed
architectures fuses information over several layers and gives the model
flexibility in how it combines information from multiple sources. We apply this
architecture to the challenge of polyp characterization and landmark
identification in endoscopy. Features from white light images are fused with
features from narrow band imaging or depth maps. This study demonstrates that
Multimodal DenseNet outperforms monomodal classification as well as other
multimodal fusion techniques by a significant margin on two different datasets.
