  Idiopathic Pulmonary Fibrosis (IPF) is an inexorably progressive fibrotic
lung disease with a variable and unpredictable rate of progression. CT scans of
the lungs inform clinical assessment of IPF patients and contain pertinent
information related to disease progression. In this work, we propose a
multi-modal method that uses neural networks and memory banks to predict the
survival of IPF patients using clinical and imaging data. The majority of
clinical IPF patient records have missing data (e.g. missing lung function
tests). To this end, we propose a probabilistic model that captures the
dependencies between the observed clinical variables and imputes missing ones.
This principled approach to missing data imputation can be naturally combined
with a deep survival analysis model. We show that the proposed framework yields
significantly better survival analysis results than baselines in terms of
concordance index and integrated Brier score. Our work also provides insights
into novel image-based biomarkers that are linked to mortality.
