  Many neurological diseases are characterized by gradual deterioration of
brain structure and function. Large longitudinal MRI datasets have revealed
such deterioration, in part, by applying machine and deep learning to predict
diagnosis. A popular approach is to apply Convolutional Neural Networks (CNN)
to extract informative features from each visit of the longitudinal MRI and
then use those features to classify each visit via Recurrent Neural Networks
(RNNs). Such modeling neglects the progressive nature of the disease, which may
result in clinically implausible classifications across visits. To avoid this
issue, we propose to combine features across visits by coupling feature
extraction with a novel longitudinal pooling layer and enforce consistency of
the classification across visits in line with disease progression. We evaluate
the proposed method on the longitudinal structural MRIs from three neuroimaging
datasets: Alzheimer's Disease Neuroimaging Initiative (ADNI, N=404), a dataset
composed of 274 normal controls and 329 patients with Alcohol Use Disorder
(AUD), and 255 youths from the National Consortium on Alcohol and
NeuroDevelopment in Adolescence (NCANDA). In all three experiments our method
is superior to other widely used approaches for longitudinal classification
thus making a unique contribution towards more accurate tracking of the impact
of conditions on the brain. The code is available at
https://github.com/ouyangjiahong/longitudinal-pooling.
