  Prognostic models aim to predict the future course of a disease or condition
and are a vital component of personalized medicine. Statistical models make use
of longitudinal data to capture the temporal aspect of disease progression;
however, these models require prior feature extraction. Deep learning avoids
explicit feature extraction, meaning we can develop models for images where
features are either unknown or impossible to quantify accurately. Previous
prognostic models using deep learning with imaging data require annotation
during training or only utilize a single time point. We propose a novel deep
learning method to predict the progression of diseases using longitudinal
imaging data with uneven time intervals, which requires no prior feature
extraction. Given previous images from a patient, our method aims to predict
whether the patient will progress onto the next stage of the disease. The
proposed method uses InceptionV3 to produce feature vectors for each image. In
order to account for uneven intervals, a novel interval scaling is proposed.
Finally, a Recurrent Neural Network is used to prognosticate the disease. We
demonstrate our method on a longitudinal dataset of color fundus images from
4903 eyes with age-related macular degeneration (AMD), taken from the
Age-Related Eye Disease Study, to predict progression to late AMD. Our method
attains a testing sensitivity of 0.878, a specificity of 0.887, and an area
under the receiver operating characteristic of 0.950. We compare our method to
previous methods, displaying superior performance in our model. Class
activation maps display how the network reaches the final decision.
