  Predicting clinical outcome is remarkably important but challenging. Research
efforts have been paid on seeking significant biomarkers associated with the
therapy response or/and patient survival. However, these biomarkers are
generally costly and invasive, and possibly dissatifactory for novel therapy.
On the other hand, multi-modal, heterogeneous, unaligned temporal data is
continuously generated in clinical practice. This paper aims at a unified deep
learning approach to predict patient prognosis and therapy response, with
easily accessible data, e.g., radiographics, laboratory and clinical
information. Prior arts focus on modeling single data modality, or ignore the
temporal changes. Importantly, the clinical time series is asynchronous in
practice, i.e., recorded with irregular intervals. In this study, we formalize
the prognosis modeling as a multi-modal asynchronous time series classification
task, and propose a MIA-Prognosis framework with Measurement, Intervention and
Assessment (MIA) information to predict therapy response, where a Simple
Temporal Attention (SimTA) module is developed to process the asynchronous time
series. Experiments on synthetic dataset validate the superiory of SimTA over
standard RNN-based approaches. Furthermore, we experiment the proposed method
on an in-house, retrospective dataset of real-world non-small cell lung cancer
patients under anti-PD-1 immunotherapy. The proposed method achieves promising
performance on predicting the immunotherapy response. Notably, our predictive
model could further stratify low-risk and high-risk patients in terms of
long-term survival.
