  Knee Osteoarthritis (KOA) is a highly prevalent chronic musculoskeletal
condition with no currently available treatment. The manifestation of KOA is
heterogeneous and prediction of its progression is challenging. Current
literature suggests that the use of multi-modal data and advanced modeling
methods, such as the ones based on Deep Learning, has promise in tackling this
challenge. To date, however, the evidence on the efficacy of this approach is
limited. In this study, we leveraged recent advances in Deep Learning and,
using a Transformer approach, developed a unified framework for the multi-modal
fusion of knee imaging data. Subsequently, we analyzed its performance across a
range of scenarios by investigating multiple progression horizons -- from
short-term to long-term. We report our findings using a large cohort
(n=2421-3967) derived from the Osteoarthritis Initiative dataset. We show that
structural knee MRI allows identifying radiographic KOA progressors on par with
multi-modal fusion approaches, achieving an area under the ROC curve (ROC AUC)
of 0.70-0.76 and Average Precision (AP) of 0.15-0.54 in 2-8 year horizons.
Progression within 1 year was better predicted with a multi-modal method using
X-ray, structural, and compositional MR images -- ROC AUC of 0.76(0.04), AP of
0.13(0.04) -- or via clinical data. Our follow-up analysis generally shows that
prediction from the imaging data is more accurate for post-traumatic subjects,
and we further investigate which subject subgroups may benefit the most. The
present study provides novel insights into multi-modal imaging of KOA and
brings a unified data-driven framework for studying its progression in an
end-to-end manner, providing new tools for the design of more efficient
clinical trials. The source code of our framework and the pre-trained models
are made publicly available.
