  Deep learning has been successfully applied to recognizing both natural
images and medical images. However, there remains a gap in recognizing 3D
neuroimaging data, especially for psychiatric diseases such as schizophrenia
and depression that have no visible alteration in specific slices. In this
study, we propose to process the 3D data by a 2+1D framework so that we can
exploit the powerful deep 2D Convolutional Neural Network (CNN) networks
pre-trained on the huge ImageNet dataset for 3D neuroimaging recognition.
Specifically, 3D volumes of Magnetic Resonance Imaging (MRI) metrics (grey
matter, white matter, and cerebrospinal fluid) are decomposed to 2D slices
according to neighboring voxel positions and inputted to 2D CNN models
pre-trained on the ImageNet to extract feature maps from three views (axial,
coronal, and sagittal). Global pooling is applied to remove redundant
information as the activation patterns are sparsely distributed over feature
maps. Channel-wise and slice-wise convolutions are proposed to aggregate the
contextual information in the third view dimension unprocessed by the 2D CNN
model. Multi-metric and multi-view information are fused for final prediction.
Our approach outperforms handcrafted feature-based machine learning, deep
feature approach with a support vector machine (SVM) classifier and 3D CNN
models trained from scratch with better cross-validation results on publicly
available Northwestern University Schizophrenia Dataset and the results are
replicated on another independent dataset.
