  While many studies have assessed the fairness of AI algorithms in the medical
field, the causes of differences in prediction performance are often unknown.
This lack of knowledge about the causes of bias hampers the efficacy of bias
mitigation, as evidenced by the fact that simple dataset balancing still often
performs best in reducing performance gaps but is unable to resolve all
performance differences. In this work, we investigate the causes of gender bias
in machine learning-based chest X-ray diagnosis. In particular, we explore the
hypothesis that breast tissue leads to underexposure of the lungs and causes
lower model performance. Methodologically, we propose a new sampling method
which addresses the highly skewed distribution of recordings per patient in two
widely used public datasets, while at the same time reducing the impact of
label errors. Our comprehensive analysis of gender differences across diseases,
datasets, and gender representations in the training set shows that dataset
imbalance is not the sole cause of performance differences. Moreover, relative
group performance differs strongly between datasets, indicating important
dataset-specific factors influencing male/female group performance. Finally, we
investigate the effect of breast tissue more specifically, by cropping out the
breasts from recordings, finding that this does not resolve the observed
performance gaps. In conclusion, our results indicate that dataset-specific
factors, not fundamental physiological differences, are the main drivers of
male--female performance gaps in chest X-ray analyses on widely used NIH and
CheXpert Dataset.
