  The rapidly emerging field of deep learning-based computational pathology has
demonstrated promise in developing objective prognostic models from histology
whole slide images. However, most prognostic models are either based on
histology or genomics alone and do not address how histology and genomics can
be integrated to develop joint image-omic prognostic models. Additionally
identifying explainable morphological and molecular descriptors from these
models that govern such prognosis is of interest. We used multimodal deep
learning to integrate gigapixel whole slide pathology images, RNA-seq
abundance, copy number variation, and mutation data from 5,720 patients across
14 major cancer types. Our interpretable, weakly-supervised, multimodal deep
learning algorithm is able to fuse these heterogeneous modalities for
predicting outcomes and discover prognostic features from these modalities that
corroborate with poor and favorable outcomes via multimodal interpretability.
We compared our model with unimodal deep learning models trained on histology
slides and molecular profiles alone, and demonstrate performance increase in
risk stratification on 9 out of 14 cancers. In addition, we analyze morphologic
and molecular markers responsible for prognostic predictions across all cancer
types. All analyzed data, including morphological and molecular correlates of
patient prognosis across the 14 cancer types at a disease and patient level are
presented in an interactive open-access database
(http://pancancer.mahmoodlab.org) to allow for further exploration and
prognostic biomarker discovery. To validate that these model explanations are
prognostic, we further analyzed high attention morphological regions in WSIs,
which indicates that tumor-infiltrating lymphocyte presence corroborates with
favorable cancer prognosis on 9 out of 14 cancer types studied.
