  In this paper, we propose a predictive regression model for longitudinal
images with missing data based on large deformation diffeomorphic metric
mapping (LDDMM) and deep neural networks. Instead of directly predicting image
scans, our model predicts a vector momentum sequence associated with a baseline
image. This momentum sequence parameterizes the original image sequence in the
LDDMM framework and lies in the tangent space of the baseline image, which is
Euclidean. A recurrent network with long term-short memory (LSTM) units encodes
the time-varying changes in the vector-momentum sequence, and a convolutional
neural network (CNN) encodes the baseline image of the vector momenta. Features
extracted by the LSTM and CNN are fed into a decoder network to reconstruct the
vector momentum sequence, which is used for the image sequence prediction by
deforming the baseline image with LDDMM shooting. To handle the missing images
at some time points, we adopt a binary mask to ignore their reconstructions in
the loss calculation. We evaluate our model on synthetically generated images
and the brain MRIs from the OASIS dataset. Experimental results demonstrate the
promising predictions of the spatiotemporal changes in both datasets,
irrespective of large or subtle changes in longitudinal image sequences.
