  Accurate survival prediction is crucial for development of precision cancer
medicine, creating the need for new sources of prognostic information.
Recently, there has been significant interest in exploiting routinely collected
clinical and medical imaging data to discover new prognostic markers in
multiple cancer types. However, most of the previous studies focus on
individual data modalities alone and do not make use of recent advances in
machine learning for survival prediction. We present Deep-CR MTLR -- a novel
machine learning approach for accurate cancer survival prediction from
multi-modal clinical and imaging data in the presence of competing risks based
on neural networks and an extension of the multi-task logistic regression
framework. We demonstrate improved prognostic performance of the multi-modal
approach over single modality predictors in a cohort of 2552 head and neck
cancer patients, particularly for cancer specific survival, where our approach
achieves 2-year AUROC of 0.774 and $C$-index of 0.788.
