  A large number of papers have introduced novel machine learning and feature
extraction methods for automatic classification of AD. However, they are
difficult to reproduce because key components of the validation are often not
readily available. These components include selected participants and input
data, image preprocessing and cross-validation procedures. The performance of
the different approaches is also difficult to compare objectively. In
particular, it is often difficult to assess which part of the method provides a
real improvement, if any. We propose a framework for reproducible and objective
classification experiments in AD using three publicly available datasets (ADNI,
AIBL and OASIS). The framework comprises: i) automatic conversion of the three
datasets into BIDS format, ii) a modular set of preprocessing pipelines,
feature extraction and classification methods, together with an evaluation
framework, that provide a baseline for benchmarking the different components.
We demonstrate the use of the framework for a large-scale evaluation on 1960
participants using T1 MRI and FDG PET data. In this evaluation, we assess the
influence of different modalities, preprocessing, feature types, classifiers,
training set sizes and datasets. Performances were in line with the
state-of-the-art. FDG PET outperformed T1 MRI for all classification tasks. No
difference in performance was found for the use of different atlases, image
smoothing, partial volume correction of FDG PET images, or feature type. Linear
SVM and L2-logistic regression resulted in similar performance and both
outperformed random forests. The classification performance increased along
with the number of subjects used for training. Classifiers trained on ADNI
generalized well to AIBL and OASIS. All the code of the framework and the
experiments is publicly available at:
https://gitlab.icm-institute.org/aramislab/AD-ML.
