  The forecasting of disease progression from images is a holy grail for
clinical decision making. However, this task is complicated by the inherent
high dimensionality, temporal sparsity and sampling irregularity in
longitudinal image acquisitions. Existing methods often rely on extracting
hand-crafted features and performing time-series analysis in this vector space,
leading to a loss of rich spatial information within the images. To overcome
these challenges, we introduce ImageFlowNet, a novel framework that learns
latent-space flow fields that evolve multiscale representations in joint
embedding spaces using neural ODEs and SDEs to model disease progression in the
image domain. Notably, ImageFlowNet learns multiscale joint representation
spaces by combining cohorts of patients together so that information can be
transferred between the patient samples. The dynamics then provide plausible
trajectories of progression, with the SDE providing alternative trajectories
from the same starting point. We provide theoretical insights that support our
formulation of ODEs, and motivate our regularizations involving high-level
visual features, latent space organization, and trajectory smoothness. We then
demonstrate ImageFlowNet's effectiveness through empirical evaluations on three
longitudinal medical image datasets depicting progression in retinal geographic
atrophy, multiple sclerosis, and glioblastoma.
