  Clinical outcome or severity prediction from medical images has largely
focused on learning representations from single-timepoint or snapshot scans. It
has been shown that disease progression can be better characterized by temporal
imaging. We therefore hypothesized that outcome predictions can be improved by
utilizing the disease progression information from sequential images. We
present a deep learning approach that leverages temporal progression
information to improve clinical outcome predictions from single-timepoint
images. In our method, a self-attention based Temporal Convolutional Network
(TCN) is used to learn a representation that is most reflective of the disease
trajectory. Meanwhile, a Vision Transformer is pretrained in a self-supervised
fashion to extract features from single-timepoint images. The key contribution
is to design a recalibration module that employs maximum mean discrepancy loss
(MMD) to align distributions of the above two contextual representations. We
train our system to predict clinical outcomes and severity grades from
single-timepoint images. Experiments on chest and osteoarthritis radiography
datasets demonstrate that our approach outperforms other state-of-the-art
techniques.
