  Previous research on the diagnosis of Bipolar disorder has mainly focused on
resting-state functional magnetic resonance imaging. However, their accuracy
can not meet the requirements of clinical diagnosis. Efficient multimodal
fusion strategies have great potential for applications in multimodal data and
can further improve the performance of medical diagnosis models. In this work,
we utilize both sMRI and fMRI data and propose a novel multimodal diagnosis
model for bipolar disorder. The proposed Patch Pyramid Feature Extraction
Module extracts sMRI features, and the spatio-temporal pyramid structure
extracts the fMRI features. Finally, they are fused by a fusion module to
output diagnosis results with a classifier. Extensive experiments show that our
proposed method outperforms others in balanced accuracy from 0.657 to 0.732 on
the OpenfMRI dataset, and achieves the state of the art.
