  Survival prediction is a major concern for cancer management. Deep survival
models based on deep learning have been widely adopted to perform end-to-end
survival prediction from medical images. Recent deep survival models achieved
promising performance by jointly performing tumor segmentation with survival
prediction, where the models were guided to extract tumor-related information
through Multi-Task Learning (MTL). However, these deep survival models have
difficulties in exploring out-of-tumor prognostic information. In addition,
existing deep survival models are unable to effectively leverage multi-modality
images. Empirically-designed fusion strategies were commonly adopted to fuse
multi-modality information via task-specific manually-designed networks, thus
limiting the adaptability to different scenarios. In this study, we propose an
Adaptive Multi-modality Segmentation-to-Survival model (AdaMSS) for survival
prediction from PET/CT images. Instead of adopting MTL, we propose a novel
Segmentation-to-Survival Learning (SSL) strategy, where our AdaMSS is trained
for tumor segmentation and survival prediction sequentially in two stages. This
strategy enables the AdaMSS to focus on tumor regions in the first stage and
gradually expand its focus to include other prognosis-related regions in the
second stage. We also propose a data-driven strategy to fuse multi-modality
information, which realizes adaptive optimization of fusion strategies based on
training data during training. With the SSL and data-driven fusion strategies,
our AdaMSS is designed as an adaptive model that can self-adapt its focus
regions and fusion strategy for different training stages. Extensive
experiments with two large clinical datasets show that our AdaMSS outperforms
state-of-the-art survival prediction methods.
