  Longitudinal studies, where a series of images from the same set of
individuals are acquired at different time-points, represent a popular
technique for studying and characterizing temporal dynamics in biomedical
applications. The classical approach for longitudinal comparison involves
normalizing for nuisance variations, such as image orientation or contrast
differences, via pre-processing. Statistical analysis is, in turn, conducted to
detect changes of interest, either at the individual or population level. This
classical approach can suffer from pre-processing issues and limitations of the
statistical modeling. For example, normalizing for nuisance variation might be
hard in settings where there are a lot of idiosyncratic changes. In this paper,
we present a simple machine learning-based approach that can alleviate these
issues. In our approach, we train a deep learning model (called PaIRNet, for
Pairwise Image Ranking Network) to compare pairs of longitudinal images, with
or without supervision. In the self-supervised setup, for instance, the model
is trained to temporally order the images, which requires learning to recognize
time-irreversible changes. Our results from four datasets demonstrate that
PaIRNet can be very effective in localizing and quantifying meaningful
longitudinal changes while discounting nuisance variation. Our code is
available at
\url{https://github.com/heejong-kim/learning-to-compare-longitudinal-images.git}
