  Time-to-event prediction, e.g. cancer survival analysis or hospital length of
stay, is a highly prominent machine learning task in medical and healthcare
applications. However, only a few interpretable machine learning methods comply
with its challenges. To facilitate a comprehensive explanatory analysis of
survival models, we formally introduce time-dependent feature effects and
global feature importance explanations. We show how post-hoc interpretation
methods allow for finding biases in AI systems predicting length of stay using
a novel multi-modal dataset created from 1235 X-ray images with textual
radiology reports annotated by human experts. Moreover, we evaluate cancer
survival models beyond predictive performance to include the importance of
multi-omics feature groups based on a large-scale benchmark comprising 11
datasets from The Cancer Genome Atlas (TCGA). Model developers can use the
proposed methods to debug and improve machine learning algorithms, while
physicians can discover disease biomarkers and assess their significance. We
hope the contributed open data and code resources facilitate future work in the
emerging research direction of explainable survival analysis.
