  Alzheimer's Disease (AD) is the most common form of dementia and often
difficult to diagnose due to the multifactorial etiology of dementia. Recent
works on neuroimaging-based computer-aided diagnosis with deep neural networks
(DNNs) showed that fusing structural magnetic resonance images (sMRI) and
fluorodeoxyglucose positron emission tomography (FDG-PET) leads to improved
accuracy in a study population of healthy controls and subjects with AD.
However, this result conflicts with the established clinical knowledge that
FDG-PET better captures AD-specific pathologies than sMRI. Therefore, we
propose a framework for the systematic evaluation of multi-modal DNNs and
critically re-evaluate single- and multi-modal DNNs based on FDG-PET and sMRI
for binary healthy vs. AD, and three-way healthy/mild cognitive impairment/AD
classification. Our experiments demonstrate that a single-modality network
using FDG-PET performs better than MRI (accuracy 0.91 vs 0.87) and does not
show improvement when combined. This conforms with the established clinical
knowledge on AD biomarkers, but raises questions about the true benefit of
multi-modal DNNs. We argue that future work on multi-modal fusion should
systematically assess the contribution of individual modalities following our
proposed evaluation framework. Finally, we encourage the community to go beyond
healthy vs. AD classification and focus on differential diagnosis of dementia,
where fusing multi-modal image information conforms with a clinical need.
