  The use of neural networks for diagnosis classification is becoming more and
more prevalent in the medical imaging community. However, deep learning method
outputs remain hard to explain. Another difficulty is to choose among the large
number of techniques developed to analyze how networks learn, as all present
different limitations. In this paper, we extended the framework of Fong and
Vedaldi [IEEE International Conference on Computer Vision (ICCV), 2017] to
visualize the training of convolutional neural networks (CNNs) on 3D
quantitative neuroimaging data. Our application focuses on the detection of
Alzheimer's disease with gray matter probability maps extracted from structural
MRI. We first assessed the robustness of the visualization method by studying
the coherence of the longitudinal patterns and regions identified by the
network. We then studied the stability of the CNN training by computing
visualization-based similarity indexes between different re-runs of the CNN. We
demonstrated that the areas identified by the CNN were consistent with what is
known of Alzheimer's disease and that the visualization approach extract
coherent longitudinal patterns. We also showed that the CNN training is not
stable and that the areas identified mainly depend on the initialization and
the training process. This issue may exist in many other medical studies using
deep learning methods on datasets in which the number of samples is too small
and the data dimension is high. This means that it may not be possible to rely
on deep learning to detect stable regions of interest in this field yet.
