  Deep learning has shown remarkable results for image analysis and is expected
to aid individual treatment decisions in health care. To achieve this, deep
learning methods need to be promoted from the level of mere associations to
being able to answer causal questions. We present a scenario with real-world
medical images (CT-scans of lung cancers) and simulated outcome data. Through
the sampling scheme, the images contain two distinct factors of variation that
represent a collider and a prognostic factor. We show that when this collider
can be quantified, unbiased individual prognosis predictions are attainable
with deep learning. This is achieved by (1) setting a dual task for the network
to predict both the outcome and the collider and (2) enforcing independence of
the activation distributions of the last layer with ordinary least squares. Our
method provides an example of combining deep learning and structural causal
models for unbiased individual prognosis predictions.
