  Recent works on Multimodal 3D Computer-aided diagnosis have demonstrated that
obtaining a competitive automatic diagnosis model when a 3D convolution neural
network (CNN) brings more parameters and medical images are scarce remains
nontrivial and challenging. Considering both consistencies of regions of
interest in multimodal images and diagnostic accuracy, we propose a novel
mutual attention-based hybrid dimensional network for MultiModal 3D medical
image classification (MMNet). The hybrid dimensional network integrates 2D CNN
with 3D convolution modules to generate deeper and more informative feature
maps, and reduce the training complexity of 3D fusion. Besides, the pre-trained
model of ImageNet can be used in 2D CNN, which improves the performance of the
model. The stereoscopic attention is focused on building rich contextual
interdependencies of the region in 3D medical images. To improve the regional
correlation of pathological tissues in multimodal medical images, we further
design a mutual attention framework in the network to build the region-wise
consistency in similar stereoscopic regions of different image modalities,
providing an implicit manner to instruct the network to focus on pathological
tissues. MMNet outperforms many previous solutions and achieves results
competitive to the state-of-the-art on three multimodal imaging datasets, i.e.,
Parotid Gland Tumor (PGT) dataset, the MRNet dataset, and the PROSTATEx
dataset, and its advantages are validated by extensive experiments.
