  Alzheimer's Disease (AD) is a progressive neurodegenerative disease. Amnestic
mild cognitive impairment (MCI) is a common first symptom before the conversion
to clinical impairment where the individual becomes unable to perform
activities of daily living independently. Although there is currently no
treatment available, the earlier a conclusive diagnosis is made, the earlier
the potential for interventions to delay or perhaps even prevent progression to
full-blown AD. Neuroimaging scans acquired from MRI and metabolism images
obtained by FDG-PET provide in-vivo view into the structure and function
(glucose metabolism) of the living brain. It is hypothesized that combining
different image modalities could better characterize the change of human brain
and result in a more accuracy early diagnosis of AD. In this paper, we proposed
a novel framework to discriminate normal control(NC) subjects from subjects
with AD pathology (AD and NC, MCI subjects convert to AD in future). Our novel
approach utilizing a multimodal and multiscale deep neural network was found to
deliver a 85.68\% accuracy in the prediction of subjects within 3 years to
conversion. Cross validation experiments proved that it has better
discrimination ability compared with results in existing published literature.
