  This article discusses how the language of causality can shed new light on
the major challenges in machine learning for medical imaging: 1) data scarcity,
which is the limited availability of high-quality annotations, and 2) data
mismatch, whereby a trained algorithm may fail to generalize in clinical
practice. Looking at these challenges through the lens of causality allows
decisions about data collection, annotation procedures, and learning strategies
to be made (and scrutinized) more transparently. We discuss how causal
relationships between images and annotations can not only have profound effects
on the performance of predictive models, but may even dictate which learning
strategies should be considered in the first place. For example, we conclude
that semi-supervision may be unsuitable for image segmentation---one of the
possibly surprising insights from our causal analysis, which is illustrated
with representative real-world examples of computer-aided diagnosis (skin
lesion classification in dermatology) and radiotherapy (automated contouring of
tumours). We highlight that being aware of and accounting for the causal
relationships in medical imaging data is important for the safe development of
machine learning and essential for regulation and responsible reporting. To
facilitate this we provide step-by-step recommendations for future studies.
